acceptodds
Under review as a conference paper at ICLR 2027

VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) struggle with fine-grained visual search, where a question concerns a small object in a high-resolution image. Existing remedies fall into two families. Training-free methods zoom in using attention or confidence scores but query the MLLM several times per example. Models trained with reinforcement learning (RL) to call cropping tools replace these queries with multi-turn rollouts but their tool calls are hard to control or interpret. To close this gap, we propose VisLens (Visual Focus via Logit Lens), which locates the target by reading the MLLM's own visual tokens through the logit lens, the projection of a hidden state through the LLM head onto the vocabulary. VisLens adds a lightweight tuned lens that maps early hidden states into the late hidden-state space, so visual tokens can be already read out from early layers. These tokens are matched to the target words in the query to produce a crop of the relevant region, which is fed back in alongside the original image to generate the final answer. The whole process, from decoding to answer, completes in a single forward pass with no repeated queries. Across three backbones, VisLens gains to points on VBench and HR-Bench and lies on the accuracy-latency Pareto frontier. It runs - faster than RL tool-use models and up to faster than training-free multi-pass search methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Search, Multimodal large language models, Logit lens

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VPmLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VisLens, a visual search method for MLLMs that localizes query-relevant regions by decoding early visual tokens through a lightweight tuned lens. The resulting crops are combined with the original image for final answering, avoiding iterative LLM-based search. Experiments across three MLLM backbones show consistent gains on fine-grained visual search benchmarks and favorable accuracy–latency trade-offs

Critical strengths
  1. Efficient and well-motivated formulation. VisLens turns early visual representations into a practical localization mechanism before LLM processing, reducing the need for iterative search or tool-use rollouts.
  2. Strong empirical validation. The method gives consistent improvements across multiple backbones and benchmarks, with supporting ablations, detector controls, and failure analyses that align well with the proposed mechanism.
Critical weaknesses

Incomplete comparison with the closest prior work. FOCUS is closely related in using internal MLLM representations for localization, but is absent from the main accuracy–latency comparison. This weakens the claimed Pareto-frontier advantage.

Questions for authors
  1. Can the authors provide a same-backbone, same-hardware accuracy–latency comparison with FOCUS?
  2. Could the authors clarify whether “single forward pass” specifically means a single LLM forward pass?
3 comments

We thank the reviewer for the positive assessment, and for finding the formulation well motivated and the empirical validation strong. We address each question below.

W1: Incomplete comparison with the closest prior work. FOCUS is closely related in using internal MLLM representations for localization, but is absent from the main accuracy–latency comparison. This weakens the claimed Pareto-frontier advantage.

We agree that FOCUS is an important related method and now discuss it more explicitly.

The key architectural distinction is where localization occurs:

  • FOCUS: Extracts relevance maps from intermediate LLM KV caches (layers 14–28), requiring an LLM pass before crop-based answering.
  • VisLens: Localizes directly from projector outputs, before any LLM layer executes, enabling crop-based answering with one LLM pass.

Because FOCUS has not released code, we cannot provide a verified same-backbone, same-hardware comparison. We have therefore clarified that our measured accuracy–latency Pareto comparison applies to the evaluated baselines, without claiming an experimentally established advantage over FOCUS.

Q2: Could the authors clarify whether "single forward pass" specifically means a single LLM forward pass?

Yes. We mean a single LLM forward pass, not a single pass through the entire multimodal pipeline.

VisLens performs target extraction, lens decoding, and crop construction before LLM inference. It requires additional vision-encoder processing for the crops and lightweight lens computation, but no second LLM pass.

We have clarified this definition at the first mention in the revised manuscript.

Dear Reviewer VPmL,

Thank you again for your review and positive assessment. In our rebuttal, we addressed your comments by:

  • Expanding the discussion of FOCUS, explaining the architectural difference, and clarifying that our accuracy–latency Pareto comparison applies only to the evaluated baselines.
  • Clarifying that "single forward pass" means a single LLM forward pass, and updating the manuscript at its first mention.

As we approach the end of the discussion period, we would appreciate hearing whether our responses have addressed your questions. We would be happy to provide any further clarifications.

Reviewer_VPmLOpenReview ↗

Response by Reviewer

Thank you for your response.

I will maintain my rating of 3. Unfortunately, I am not very familiar with this research area, so my confidence score is 2.

I am also somewhat puzzled as to why several papers assigned to me for ICLR this year fall outside my areas of expertise. I would appreciate it if paper assignments could be more closely aligned with my broader research area, as they have been in previous years.

Reviewer_u3VNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VisLens, which performs visual search by directly decoding visual representations through a lightweight tuned logit lens and matching the resulting tokens to object names extracted from the question. It constructs crops from the matched regions and provides them together with the original image for answering. To ensure a single inference round, it uses pre-LLM vision representations to avoid an additional LLM pass. Experiments report accuracy-latency improvements across three backbones, with supplementary evaluations on standard VQA and video retrieval.

Critical strengths
  1. The method provides a simple approach to reducing the inference overhead of iterative visual search.
  2. The source-layer and crop-construction ablations are useful. Qualitative examples make localization decisions inspectable, and the paper documents failures involving diagrams and repeated objects.
Critical weaknesses
  1. The practical motivation needs validation against newer multimodal backbones. The fine-grained perception abilities in high-resolution images have been developing rapidly and have greatly improved. The experiments mainly use LLaVA-OneVision, Qwen2.5-VL, and InternVL3, which are kind of 'outdated '. The paper does not establish whether explicit crop-based search remains necessary or cost-effective with newer models. A recent comparator such as Qwen3.5-9B, evaluated both without search and with VisLens, would be informative.
  2. The semantic interpretation of visual-token logit-lens outputs needs stronger validation. Visual positions are not directly supervised to predict local object names or receive any direct textual supervision through the LM head. They receive indirect training signals, but this does not guarantee faithful vocabulary-level interpretations. Distilling final-layer visual-position logits transfers that readout without independently validating its semantics. This concern is particularly relevant to pre-LLM projector outputs, which lack LLM-level refinement.
  3. Noun-based matching and one crop per phrase restrict practical applicability. The pipeline supports multiple named targets, but does not explicitly resolve indirect references or distinguish instances of the same category. In “What color is the object to the left of the table?”, the named noun identifies the reference object rather than the target; the implementation also filters generic words such as “object.” Multiple category matches may produce an ambiguous merged crop or exceed the area cap. Systematic evaluation on relational descriptions, indirect references, and repeated objects is needed to establish coverage beyond explicitly named targets.
  4. The video evaluation is too limited to establish broader video utility. Video offers a potentially stronger motivation for selective processing, but the current evidence is limited to synthetic needle retrieval on VNBench.
Questions for authors
  1. External detectors return no box on roughly 40% of queries at their default thresholds. How do the results change after threshold tuning or matching crop recall and crop-area budgets?
  2. What are the actual visual-token and crop counts per question?
4 comments

We thank the reviewer for a detailed and constructive review, and for finding the ablations, qualitative examples, and failure analysis useful. We respond to each concern below.

W1: The practical motivation needs validation against newer multimodal backbones. The fine-grained perception abilities in high-resolution images have been developing rapidly and have greatly improved. The experiments mainly use LLaVA-OneVision, Qwen2.5-VL, and InternVL3, which are kind of 'outdated '. The paper does not establish whether explicit crop-based search remains necessary or cost-effective with newer models. A recent comparator such as Qwen3.5-9B, evaluated both without search and with VisLens, would be informative.

We agree that newer backbones should be evaluated. Although stronger models improve visual understanding, small-target recognition remains constrained by visual token allocation, especially under fixed resolution and token budgets.

To test whether VisLens remains beneficial, we extend evaluation to two newer backbones, InternVL3.5-8B and LLaVA-OneVision-2-8B-Instruct (LLaVA-OV-2-8B), using V*Bench and HR-Bench.

ModelV*BenchHR-Bench 4KHR-Bench 8K
InternVL3.5-8B71.265.961.3
InternVL3.5-8B + VisLens78.074.866.5
LLaVA-OV-2-8B85.8673.1265.62
LLaVA-OV-2-8B + VisLens88.4877.6274.25

VisLens improves InternVL3.5-8B by +6.8, +8.9, and +5.2 points, respectively, and LLaVA-OV-2-8B by +2.62, +4.50, and +8.63 points, demonstrating that explicit localization remains useful on newer backbones.

We retain the original backbones to preserve controlled comparisons with existing training-free and RL-based methods.

W2: The semantic interpretation of visual-token logit-lens outputs needs stronger validation. Visual positions are not directly supervised to predict local object names or receive any direct textual supervision through the LM head. They receive indirect training signals, but this does not guarantee faithful vocabulary-level interpretations. Distilling final-layer visual-position logits transfers that readout without independently validating its semantics. This concern is particularly relevant to pre-LLM projector outputs, which lack LLM-level refinement.

We agree that visual-token logits should not be assumed semantically faithful without validation.

Table 5 in the revised manuscript provides ground-truth validation on COCO small objects (0.01–1.5% image area):

  • 56.7% token detection: The tuned lens decodes the target object name or a synonym at its annotated position.
  • 64.1% GT coverage: Generated crops spatially overlap the annotated target regions.
  • Comparable to final-layer readout: Coverage differs by less than 0.015 despite using only pre-LLM features.

These measurements independently assess semantic agreement and spatial localization, rather than relying on downstream VQA accuracy.

Our claim is not that every decoded token faithfully describes its visual patch, but that the lens exposes a measurable semantic localization signal sufficient for visual search.

Importantly, the lens uses self-distillation without object-level annotations, while keeping the backbone frozen. Prior logit-lens studies [1,2] further support the presence of spatially grounded semantics in visual-token readouts.

We have moved Table 8 of the original submission into the main paper (now Table 5).

[1] Neo et al. Towards Interpreting Visual Information Processing in Vision-Language Models. ICLR 2025

[2] Jiang et al. Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations. ICLR 2025

W3: Noun-based matching and one crop per phrase restrict practical applicability. The pipeline supports multiple named targets, but does not explicitly resolve indirect references or distinguish instances of the same category. In "What color is the object to the left of the table?", the named noun identifies the reference object rather than the target; the implementation also filters generic words such as "object." Multiple category matches may produce an ambiguous merged crop or exceed the area cap. Systematic evaluation on relational descriptions, indirect references, and repeated objects is needed to establish coverage beyond explicitly named targets.

We acknowledge these limitations.

Indirect references. VisLens targets explicitly named objects, consistent with the construction of V*Bench and HR-Bench. Queries such as “the object to the left of the table” are outside the evaluated scope.

Relational reasoning. Table 6 shows improvements of +12.1 to +13.1 points for single-target questions and +6.6 to +10.5 points for two-target spatial questions, indicating reduced benefits when relationships between regions are essential.

Repeated instances. Multiple instances may share a crop or be excluded by the area cap. This limitation is documented in Fig. 8 and Table 7.

We have clarified these boundaries and identified relation-aware localization and instance separation as future extensions.

W4: The video evaluation is too limited to establish broader video utility. Video offers a potentially stronger motivation for selective processing, but the current evidence is limited to synthetic needle retrieval on VNBench.

We agree that VNBench alone cannot establish general video-QA utility.

Our experiment demonstrates zero-shot transfer of the image-trained lens to frame-level visual search, yielding improvements of +4.3 to +11.0 points across three backbones.

We have explicitly positioned this result as preliminary and clarified two missing capabilities: adaptive frame selection without answer-option guidance and localization of targets that cannot be expressed as object names.

Q1: External detectors return no box on roughly 40% of queries at their default thresholds. How do the results change after threshold tuning or matching crop recall and crop-area budgets?

We additionally sweep detector thresholds under the same crop-and-answering protocol as Table 4, with LLaVA-OV-7B as the answering backbone on V*Bench.

Detector, threshold (LLaVA-OV-7B)Qs with a boxGT coverageCrop inclusion (all objects)Accuracy
SAM3, 0.50 (default)58.1%61.10.13574.3
SAM3, 0.25 (best acc.)71.2%59.60.15976.4
SAM3, 0.1079.1%63.00.19275.9
Grounding-DINO, 0.50 (default)60.3%59.90.10672.3
Grounding-DINO, 0.35 (best acc.)71.5%57.30.17674.4
Grounding-DINO, 0.0597.9%48.20.26571.2
VisLens99.5%86.30.83383.2

Lower thresholds increase crop frequency but do not reliably improve localization.

Even at their best accuracy thresholds, SAM3 (76.4%) and Grounding-DINO (74.4%) remain below VisLens (83.2%). VisLens also achieves substantially higher crop inclusion (0.833 vs. ≤0.265).

Thus, threshold tuning alone does not close the localization or accuracy gap.

Q2: What are the actual visual-token and crop counts per question?

We report actual visual-token counts before and after adding VisLens crops.

BackboneBenchmarkBaseline tokens, mean (p90)VisLens tokens, mean (p90)Ratio
LLaVA-OV-7BV*7,320 (7,332)13,368 (18,074)1.8×
HR-4K7,316 (7,371)19,894 (29,159)2.7×
HR-8K7,303 (7,329)20,278 (29,155)2.8×
Qwen2.5-VL-7BV*1,990 (2,028)2,985 (3,843)1.5×
HR-4K2,014 (2,025)5,131 (7,908)2.5×
HR-8K1,996 (2,016)5,300 (7,959)2.7×
InternVL3-8BV*2,008 (3,328)4,321 (6,400)2.2×
HR-4K2,509 (2,560)7,045 (9,754)2.8×
HR-8K2,025 (2,560)6,742 (9,754)3.3×

VisLens increases visual-token consumption by 1.5–3.3×, particularly on high-resolution benchmarks, where native-resolution crops introduce additional tokens.

However, these tokens are input tokens, processed in parallel within a single LLM prefill. RL-based methods on the same Qwen2.5-VL-7B backbone instead spend their cost on generated tokens, which are decoded one at a time across multiple model calls:

Method (Qwen2.5-VL-7B)Generated tokens, V* (median†)HR-4KHR-8KModel calls per questionCrops per question (V* / 4K / 8K)Questions with no crop
DeepEyes v1207214219~2.01.00 / 1.02 / 0.99~1%
DeepEyesV2275266275~2.251.40 / 1.12 / 1.174–19%
Mini-o3257274323~3.1–3.62.32 / 2.14 / 2.5513–20%
VisLens22211.44 / 1.96 / 1.960.5%

† We report medians because DeepEyesV2 has rare runaway generations.

All methods use a similar number of crops per question; the difference is the cost of choosing them. VisLens produces one crop per object phrase before the LLM runs, so it decodes only the answer in one call, over 100× fewer generated tokens than the RL-based methods. This is why it adds only 0.26–0.46 s to the plain Qwen2.5-VL answer. The trade-off is some accuracy on V* (76.4 vs. 80.6–86.4), in exchange for no training and a single LLM pass.

Dear Reviewer u3VN,

Thank you again for your review and constructive feedback. In our rebuttal, we addressed your comments by:

  • Evaluating VisLens on two newer backbones, InternVL3.5-8B and LLaVA-OV-2-8B, on V*Bench and HR-Bench.
  • Pointing to the ground-truth validation of the lens's semantics and localization on COCO small objects, which we moved into the main paper.
  • Clarifying the scope of VisLens with respect to indirect references, relational questions, and repeated instances.
  • Positioning the VNBench video results as preliminary and stating the missing capabilities.
  • Sweeping the confidence thresholds of SAM3 and Grounding-DINO under the same crop-and-answering protocol.
  • Reporting actual visual-token and crop counts per question, with a cost comparison against RL-based methods.

As we approach the end of the discussion period, we would appreciate hearing whether our responses have addressed your questions. We would be happy to provide any further clarifications.

Reviewer_2GquOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In fine-grained visual search task, a small region relevant to the input text query needs to be cropped and zoom-in on. To locate this region, the authors propose VisLens, which first maps visual features into text tokens using small MLP and decoding head. Then it constructs crop region through text matching and clustering. VisLens achieves efficient inference by performing this cropping before visual features are processed by LLM.

Critical strengths

VisLens uses a logit lens to match visual tokens with textual semantics through only an additional small MLP projection and the decoding head, thereby indentifying target crop regions. This is efficient in terms of inference time compared with training-free methods that require multiple LLM inferences or RL tool-use methods that require textual reasoning to decide where to crop.

Critical weaknesses

There is no direct comparison of crop quality across different cropping methods.

Crop quality is evaluated only for the source layer comparison within VisLens (Table 8). Therefore, it does not demonstrate that VisLens identifies better crop region than other methods.

Using final layer features as the teacher representation for mapping visual features does not sufficiently justified.

Although these final layer features are aligned with the decoding head, this does not establish that they are suitable target for mapping to use logit lens. Prior work [1] reports that middle-to-late layers recover relevant textual semantics through the logit lens, however applying the logit lens to later layer features yiels tokens such as punctuation marks or conjunctions rather than sematically relevant texts. Nevertheless, the authors use final layer features as the teacher representation for mapping visual features before LLM, without comparing alternative target layers.

[1] Jiang et al. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. CVPR 2025

Questions for authors
  1. Can the authors compare crop quality across VisLens and the baseline methods using metrics such as GT coverage (reported in Table 8)?

  2. Can the authors provide an ablation using different target layer as teacher representations for the tuned lens?

  3. Does the merging procedure in VisLens assume that at most one object in the image matches each target phrase?

5 comments

We thank the reviewer for a careful review, and for noting the inference efficiency of VisLens. We respond to each concern below with new results.

W1: There is no direct comparison of crop quality across different cropping methods. Crop quality is evaluated only for the source layer comparison within VisLens (Table 8). Therefore, it does not demonstrate that VisLens identifies better crop region than other methods.

We agree that direct localization metrics complement downstream accuracy. We therefore evaluate crop quality on V*Bench using 245 ground-truth objects across 191 questions, comparing the crops actually generated by each method. Objects in questions without a crop count as misses.

MethodBackboneCrop inclusion (all objects)GT coverageCrop / image areaAcc. (own)
VisLensLLaVA-OV-7B0.83386.321.9%83.2
ViCropLLaVA-OV-7B0.57662.34.4%76.4
ZoomEyeLLaVA-OV-7B0.71095.210.9%91.1
VisLensQwen2.5-VL-7B0.66167.821.7%76.4
Mini-o3Qwen2.5-VL-7B0.69887.112.0%81.7
DeepEyesV2Qwen2.5-VL-7B0.11430.50.7%80.6
DeepEyes v1Qwen2.5-VL-7B0.10256.10.15%86.4

Key findings:

  • Training-free localization: VisLens improves inclusion over ViCrop from 0.576 to 0.833. ZoomEye achieves higher GT coverage but crops only 139/191 questions, reducing overall inclusion to 0.710, with approximately 11× higher latency.
  • RL-based methods: DeepEyes variants achieve high answering accuracy despite low crop inclusion (~0.10). Mini-o3 reaches 0.698 inclusion using multiple sequential zooms per question.

These results establish that VisLens provides stronger single-stage target localization, while also showing why crop quality must be evaluated independently of downstream accuracy.

W2: Using final layer features as the teacher representation for mapping visual features does not sufficiently justified. Although these final layer features are aligned with the decoding head, this does not establish that they are suitable target for mapping to use logit lens. Prior work [1] reports that middle-to-late layers recover relevant textual semantics through the logit lens, however applying the logit lens to later layer features yiels tokens such as punctuation marks or conjunctions rather than sematically relevant texts. Nevertheless, the authors use final layer features as the teacher representation for mapping visual features before LLM, without comparing alternative target layers.

[1] Jiang et al. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. CVPR 2025

We agree that final-layer selection requires empirical justification. We therefore sweep raw logit-lens readouts across layers, keeping the remaining pipeline fixed.

Lens (Qwen2.5-VL-7B, V*Bench)Phrases in top-10Crop inclusionAccuracy
Raw L-1 (projector output)22.9%0.31864.4%
Raw L718.9%0.33163.9%
Raw L1416.0%0.30265.4%
Raw L1816.0%0.25363.4%
Raw L2124.0%0.26562.3%
Raw L2350.9%0.44968.1%
Raw L2573.1%0.64573.8%
Raw L27 (final)76.0%0.65376.4%
Tuned lens, L-1 → L27 (ours)74.2%0.66176.4%
Lens (LLaVA-OV-7B, V*Bench)Phrases in top-10Crop inclusionAccuracy
Raw L-1 (projector output)50.9%0.35171.7%
Raw L770.9%0.44174.9%
Raw L1466.9%0.44973.3%
Raw L1865.1%0.46172.8%
Raw L2181.1%0.61674.9%
Raw L2392.0%0.78079.1%
Raw L2596.0%0.80882.7%
Raw L27 (final)96.0%0.80482.7%
Tuned lens, L-1 → L27 (ours)93.8%0.83383.2%

On both backbones, late-layer readouts provide substantially stronger query-relevant semantics than middle-layer readouts. The tuned lens recovers comparable localization performance directly from pre-LLM features, without executing the LLM.

Unlike the late-layer punctuation effects reported by [1], we observe strong semantic matching at late layers on both evaluated architectures, suggesting architecture- or setting-dependent behavior.

We clarify that this experiment compares teacher-layer readouts, rather than retraining separate tuned lenses for each target layer. The final layer is selected because it consistently provides strong semantic signals and is directly aligned with the LM head.

[1] Jiang et al. Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. CVPR 2025

Q3: Does the merging procedure in VisLens assume that at most one object in the image matches each target phrase?

No. VisLens supports multiple matches but produces one crop per phrase.

Matched cells are grouped into 8-connected components and merged by score under a strict 20% image-area cap. Nearby instances can share a crop; distant instances may be excluded.

This limitation is already illustrated in Fig. 8 and the remote-sensing results in Table 7. We have clarified it in the method and limitations sections of the revised manuscript.

Note on Informativeness 1. Unlike FOCUS, which derives relevance maps from intermediate LLM KV caches and therefore requires an LLM pass before cropping, VisLens localizes regions directly from pre-LLM visual features. Unlike prior logit-lens methods focused on interpretation and hallucination detection, VisLens trains a lightweight lens to perform query-guided localization before any LLM layer executes, enabling visual search with a single LLM pass.

Dear Reviewer 2Gqu,

Thank you again for your review and constructive feedback. In our rebuttal, we addressed your comments by:

  • Adding a direct crop-quality comparison across cropping methods on V*Bench (crop inclusion, GT coverage, and crop area), covering both training-free and RL-based baselines.
  • Sweeping raw logit-lens readouts across layers on two backbones to justify the final layer as the teacher representation, and discussing the difference from the findings of Jiang et al.
  • Clarifying how the merging procedure handles multiple instances of the same phrase, and stating the resulting limitation in the method and limitations sections.
  • Clarifying how VisLens differs from FOCUS and from prior logit-lens methods.

As we approach the end of the discussion period, we would appreciate hearing whether our responses have addressed your questions. We would be happy to provide any further clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.