acceptodds
Withdrawn from ICLR 2027

Rewarding What Matters: Human-Inspired Visual Search for Fine-Grained MLLM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have made substantial progress in visual understanding, yet they still struggle to resolve small but answer-critical details in high-resolution images. Exhaustively processing those images can preserve such details but incurs substantial visual-token cost, motivating selective visual inspection. In contrast, human visual search efficiently prioritizes information through complementary cognitive mechanisms: bottom-up processes that suppress irrelevant distractions and top-down cognitive control that directs attention toward task-relevant information. Inspired by these mechanisms, we introduce SAGE (Surprise-And-Goal-Guided Exploration), a training-free visual search method with a bottom-up surprise module and a top-down relevance module. The surprise module identifies more informative visual tokens relative to their surrounding context, while the relevance module selects regions that are most semantically relevant to the task query. Together, these modules enable efficient filtering of redundant visual content while preserving task-critical evidence. Experiments across four MLLM families demonstrate that our method consistently improves accuracy over ZoomEye, while requiring fewer visual tokens.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-grained MLLM, Visual Search, Human-centric AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bc52OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a training-free visual search method (SAGE) to solve fine-grained details perception in high-resolution images. SAGE consists of a bottom-up surprise module and a top-down relevance module. The former organizes high-resolution image into overlapping square tiles and identifies more informative visual tokens relative to their surrounding context, while the latter selects regions that are most semantically relevant to the task query. The outputs of two modules are used to guides crop proposals, which are reranked using target-existence probabilities. Final answering uses the global image and selected local evidence. Experiments with four MLLM backbones on three high-resolution benchmarks demonstrate the proposed SAGE consistently improves accuracy.

Critical strengths
  1. The paper attempts to use the bottom-up and top-down search mechanisms in the human visual search process to solve the problem of high-resolution image perception, and designs two modules to implement it. The experimental results indicate that this type of person search mechanism has high search efficiency.
  2. SAGE outperforms both the corresponding base model and ZoomEye in overall accuracy across all backbone–benchmark combinations. With LLaVA-OV-7B, it achieves 94.8%, 73.4%, and 70.3% on V* Bench, HR-Bench-4K, and HR-Bench-8K, respectively. This breadth strengthens the practical empirical contribution.
Critical weaknesses
  1. The two core modules in SAGE simply adapt the designs from existing methods to the visual search mechanism. SURGE already combines surprise-guided computation with optional query-relevance filtering in video understanding task and FOCUS already uses internal visual–text similarity to construct relevance maps, generate crop proposals, and re-rank them using target-existence confidence. In addition, SURGE defines surprise through temporal prediction errors, whereas SAGE measures deviation from a cross-tile mean whose interpretation as expected visual content is unclear in heterogeneous scenes. This deviation reflects feature distinctiveness, which does not inherently indicate task-critical evidence or favor small objects. Although the ablation suggests incremental utility, it does not establish the suitability of this reference or substantiate the proposed interpretation of spatial surprise.
  2. Figure 2 in the paper appears to be an AI-generated image that has not been manually edited and contains multiple non-standard contents. Firstly, the Re rank selected patches in component "3. Cognitive Filtering" are very unclear, making it impossible to confirm whether they are the content of the original image, and the sorting score is completely unreadable. Secondly, the icon sizes vary and are not strictly aligned. Thirdly, the image resolution is low, making it difficult to distinguish detailed information.
  3. Lack of generalization experiments on other high-resolution benchmarks, such as MME-RealWorld-Lite [1] and ZoomBench [2].
  4. Hyperparameter sensitivity is unclear. Important choices include tile count, overlap ratio, IoU threshold, layer averaging set L, and the existence-verification prompt. There is little or no sensitivity analysis.
  5. The paper’s writing requires further refinement. For example, line 38-41 and line 90-91.

[1] Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?

[2] Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception.

Questions for authors
  1. What is the core technical contribution of the proposed method compared to existing methods SURGE and FOCUS? Has it been verified through experiments?
  2. How are surprise scores and relevance scores fused? Are they multiplied, added, normalized, or weighted? How are their scales matched?
  3. How exactly is SAGE applied to multi-view inputs? Are the two views encoded jointly as a longer video sequence? How are candidates selected across views, and how is the final crop composed?
Reviewer_Deq8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on fine-grained perception and high-resolution small object perception. Building on existing works like ZoomEye and FOCUS, the authors posit that humans possess a bottom-up, saliency-guided observation mechanism that is absent in current training-free methods, and they seek to demonstrate the effectiveness of this approach.

Critical strengths

None

Critical weaknesses
  1. Missing closely related works: A substantial body of tightly coupled prior work appears to be conspicuously missing from the paper. Relevant and effective methods such as ZwZ, SenseNova-MARS, and HiDe, approaches whose strong performance the proposed method might struggle to surpass, have been entirely overlooked in the comparisons.
  2. Casual presentation and missing critical formulation: The writing is somewhat sloppy, and key details are omitted. For example, regarding the statement "The reference μj\mu_j is estimated from the current image in feature space", the key variable μj\mu_j, which lies at the core of the only relatively unique bottom-up component, lacks a formal and rigorous mathematical definition. Such a vague description makes it difficult to understand the proposed method.
  3. Weak motivational justification: While humans indeed possess bottom-up visual attention mechanisms, the paper provides insufficient evidence or theoretical grounding to establish a meaningful connection between its proposed bottom-up pipeline and actual human perceptual mechanisms.
  4. Practical utility concerns regarding efficiency: Training-free approaches typically incur substantial computational and runtime overhead. In terms of practical efficiency and real-world applicability, I suspect this method falls far short of existing alternatives like ZwZ.
Questions for authors
  1. Could the authors explicitly clarify how the reference variable μj\mu_j is computed, and provide concrete evidence (e.g., targeted ablation studies) to validate the effectiveness of the proposed bottom-up mechanism?
  2. Please provide detailed runtime and computational latency benchmarks (e.g., FPS / inference time per image) comparing against other baseline methods.
  3. Experimental comparisons against the aforementioned missing baselines (e.g., ZwZ, SenseNova-MARS, HiDe) should be included to fairly assess the competitive standing of this work.
Reviewer_DKmNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to find small, answer-critical details in high-resolution images. It proposes SAGE, a training-free method. SAGE cuts the image into tiles, encodes the tiles together through the model's native video interface, and builds a routing heatmap from two signals: bottom-up surprise and top-down relevance to the question. It picks a few candidate windows, checks them, and adds the best one to the global image. The paper reports gains over the backbone and over ZoomEye on several benchmarks, with lower time and token cost, and adds a small new multiview benchmark.

Critical strengths

The gains are consistent. SAGE beats ZoomEye in every block of Table 1, across four current backbones. SAGE is also more efficient both in wall-clock time and VRAM. Table 5 is informative — 60% of the remaining errors are correct-crop-wrong-answer — so the paper scopes its own claim to evidence selection.

Critical weaknesses
  1. The gain of the new part is limited. In Table 3, top-down relevance alone reaches 92.67/70.75, and full SAGE reaches 94.76/73.38. The "human-inspired" surprise module adds only 2 points. The paper does not show a question type where surprise helps much more, so the framing oversells this module.

  2. The cost win mixes two effects. Video frames cost 196 tokens against 729 per image (Section 3.2). So Table 2's 70.7% token cut mixes smarter routing with cheaper per-tile encoding. A control with standard multi-image encoding would show what the method itself saves.

Questions for authors

Please refer to above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.