acceptodds
Withdrawn from ICLR 2027

EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained visual perception enables vision–language models to distinguish subtle attributes and ground their answers in visual evidence. In high-resolution scenes, processing the whole image at greater resolution spends visual tokens on irrelevant content, while isolated crops can lose the context needed to interpret the selected evidence. We introduce EviViT, a lightweight attachment that learns where a pretrained vision transformer should acquire detail. Human visual-search traces supervise a question-conditioned evidence density, which guides regional re-reading from the original pixels and the allocation of visual tokens. A sparse, coordinate-aware bridge then connects the regional features to the global scene, allowing the host to interpret precise evidence in context. Learned with the host backbone frozen, the attachment serves both the base model and compatible post-trained descendants without refitting. Experiments across nine hosts show consistent gains in average fine-grained accuracy. Matched-budget comparisons further show that EviViT outperforms global-only processing at every tested token ceiling while using fewer visual tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Fine-Grained Perception, Visual Token Allocation, Human Visual Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qUosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Fine-grained VQA on high-resolution scenes involves a trade-off. Global resizing loses fine details. Uniformly raising the resolution of the whole image wastes computation on irrelevant background. Active-cropping methods (V*, Mini-o3) need multiple LLM forward passes, which makes inference expensive and ties evidence acquisition to the answering policy.

The paper proposes EviViT, a lightweight learnable module inserted into the intermediate layers of a frozen ViT. A question-conditioned "evidence density" is distilled from human visual-search trajectories and supervises the Prompt-conditioned Token–Evidence Aligner (PTEA), which predicts where high-resolution detail should be acquired. A planner then selects at most two decisive regions plus one complementary contextual region and re-encodes them from the original pixels. A coordinate-aware Sparse Bridge fuses the resulting features with the global tokens. The original ViT, merger, and LLM stay frozen throughout. The attachment can be reused across compatible post-trained models (e.g., from Qwen3-VL to its Qwen3.5/Vision-OPD descendants).

Experiments cover nine hosts (4B–9B). EviViT improves the seven-benchmark average by 1.98–7.40 points. Under matched token budgets from 1K to 20K, it outperforms global-only processing across the board while using only 90–97% of the tokens. General capabilities (MMBench/MMStar/Visual-CoT) are largely preserved, and the method is compatible with language-side LoRA SFT.

Critical strengths

Novel perspective. Unlike the iterative cropping of V*/Mini-o3 and the distillation approaches of ZwZ/Vision-OPD, this work treats "evidence acquisition" as a separate problem and implements it as a standalone attachment on a frozen ViT.

Human-trajectory supervision. Supervision comes from complete, real human search trajectories. The full-trace vs. final-box comparison in Table 5 (VP-Hard +3.77) directly supports this core design choice.

Rigorous controls and thorough ablations.The location controls in Table 8 (random translation, cross-image shuffling) show that the gains come from "choosing the right locations." Table 14 shows that the deployed crop regions cover only 12.39% of the image area yet achieve an average coverage of 73.69% of the official targets. Table 9 uses per-question rescue/loss statistics and McNemar tests to rule out "random answer flipping" as an explanation.

Good reproducibility and practicality. Training time is short, the host is fully frozen, and the attachment transfers directly to post-trained derivatives that share the same vision tower.

Critical weaknesses

The appendix experiments (Table 13, pooled) undermine the core claims. (a) The Sparse Bridge provides no gain over simply feeding the crops as multi-image input. The contribution that "coordinate-aware fusion helps understand details in scene context" is not supported by the data; its only remaining role is engineering convenience at the interface level. (b) Cropping from the already-resized global image performs exactly the same as "re-reading from the original pixels at native detail." This contradicts the central claim in the Introduction that "selecting encoded tokens cannot recover detail lost during initial resizing."

Missing key comparisons. Comparing only against global-only is insufficient. Under the same host and the same token budget, the method should be compared at least against training-free question-guided cropping (ViCrop, ZoomEye, DC², etc.) and region selection based on VLM internal attention or relevance (FOCUS, etc.).

In-distribution advantage and insufficient evidence of generalization. The PTEA trajectories and the Bridge's answer cross-entropy both come from the VisualProbe training set, and the largest gains appear on VisualProbe Medium/Hard (average +8.2 and +8.6). Gains on HR-Bench are much smaller. In Table 9, the McNemar p-values for the Easy and Hard tiers are not significant on several hosts.

Questions for authors

Regarding Table 13: cropping from the resized global image yields exactly the same result as re-reading from the original image (66.57). How do you reconcile this with the core claim of "recovering detail lost to resizing"? Please rerun this control in a setting where the global view is heavily downscaled (e.g., 1K/2K budgets).

Under the same host and the same token budget, how does EviViT compare with "Qwen3-VL's own grounding → crop → multi-image input" and with ViCrop/ZoomEye?

Do the VisualProbe training set, its test set, and V* overlap in image sources? What exactly does the "186-image overlap" in the appendix refer to?

Reviewer_7fC3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EviViT, a lightweight attachment for improving fine-grained visual perception in vision-language models (VLMs). The key idea is to use human visual-search traces to learn where additional visual resolution should be allocated for a given question. Specifically, the method converts cursor movements, zoom windows, and final evidence annotations into a question-conditioned spatial evidence distribution. A Prompt-conditioned Token–Evidence Aligner (PTEA) predicts this distribution, which guides the selection of decisive regions and complementary context regions. These regions are re-read from the original image at higher resolution and integrated with global visual features through a coordinate-aware Sparse Bridge. The host vision encoder, merger, and language model remain frozen during training. Experiments across nine foundation and post-trained hosts show consistent improvements on fine-grained perception benchmarks. The method also improves the accuracy–token trade-off under matched token budgets and demonstrates that full search-trace supervision provides additional benefits over supervision using only final evidence boxes.

Critical strengths
  • A practical approach to fine-grained visual perception. The paper addresses an important limitation of VLMs: resizing high-resolution images can remove small but critical visual details. Rather than increasing the resolution of the entire image or relying on repeated language-model crop requests, EviViT selectively acquires additional details while preserving the global scene.
  • A meaningful use of human visual-search traces. The paper goes beyond using only final evidence boxes by incorporating intermediate search regions, zoom operations, and the final search branch.
  • Strong and consistent experimental performance. EviViT achieves improvements over the baseline models across most evaluated settings, with gains observed on multiple fine-grained perception benchmarks and across different model sizes. The results suggest that its evidence-guided visual acquisition strategy is broadly effective rather than being limited to a specific model or dataset.
Critical weaknesses
  • The human-search supervision dataset is relatively limited. EviViT is trained using approximately 1,144 annotated search examples collected from members of the research team. Although the results are promising, the paper does not sufficiently establish how robust the learned evidence allocation is to differences in annotators, search strategies, and question types. The effectiveness of the method may depend on the quality and diversity of these traces. Evaluation with a larger and more diverse annotation set would strengthen the generalization claims.
  • The improvements do not uniformly preserve all general capabilities. While the paper reports broadly stable performance on general multimodal benchmarks, some configurations exhibit noticeable regressions. For example, Qwen3-VL-4B loses 2.01 points on Visual-CoT, and the post-trained ZwZ-8B model loses 3.13 points on MMStar in Appendix Table 11. These results suggest that integrating additional regional features can affect how different host models interpret visual information. A more systematic analysis of these regressions and their causes would clarify the compatibility limitations of EviViT.
  • Lack of clarity in the method description. The methodology is difficult to follow, particularly the sections "From Search Events to Evidence Density" and "Native Re-Reading and Sparse Fusion." The explanations do not clearly convey how the proposed components work or how they interact with each other. In addition, the framework figures are not sufficiently informative, making it difficult to understand the overall pipeline and the roles of individual components. More intuitive explanations and clearer illustrations are needed to improve the readability of the paper.
  • Insufficient analysis of performance degradation. Table 4 shows that EviViT leads to performance drops on certain general-capability benchmarks, such as Visual-CoT for Qwen3-VL-4B and MMBench for Qwen3.5-9B. However, the paper does not sufficiently investigate the causes of these regressions. Further analysis is needed to understand how EviViT affects the host models' general capabilities and under what conditions the proposed method may be detrimental.
Questions for authors

please refer to the weakenss.

Reviewer_GrFBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EviViT is a lightweight attachment to a frozen Qwen-VL vision transformer. Supervised by 1,144 author-collected VisualProbe search traces, it predicts a question-conditioned evidence density, re-reads at most two decisive regions and one context region from the original pixels, and fuses them into the global grid with a coordinate-aware sparse bridge. The host remains frozen, and the attachment is reused without refitting on post-trained models that share its visual stack.

Across nine hosts, the gain in the seven-benchmark average ranges from 1.98 to 7.40 points, with larger gains on VisualProbe Medium and Hard (Table 1). On Qwen3-VL-4B and Qwen3-VL-8B, under matched ceilings from 1K to 20K tokens, EviViT outperforms global-only processing at every ceiling while using 90-97% as many visual tokens (Table 2).

Critical strengths
  1. The matched-budget comparison is the result that most supports the paper's claim. On Qwen3-VL-4B and Qwen3-VL-8B, EviViT is more accurate than global-only processing at every ceiling from 1K to 20K tokens. With a 4K ceiling, the 8B model reaches 68.87 accuracy using 3.62K tokens, exceeding the global-only result of 68.27 at a 20K ceiling, where 13.31K tokens are realized.

  2. The attachment is reusable, and it remains useful after the language model continues to train. Fit on a frozen foundation host, the evidence allocator is applied without refitting to compatible post-trained hosts, including ZwZ, Vero, and Vision-OPD; the paper reports a higher seven-benchmark average for all nine hosts. On Qwen3-VL-8B, under a matched LoRA schedule of 10,000 examples, EviViT+SFT retains a lead of 6.80 points on VP-All and 7.34 points on ZoomBench over Base+SFT.

Critical weaknesses
  1. The experiments do not test the comparison that the introduction relies on. Section 1 presents EviViT as an alternative to active search, including V* and Mini-o3, but every locally evaluated result compares a host only with the same host plus EviViT under global resizing. Without one matched-budget comparison against such a method, the claimed advantage over that line of work is not established.

  2. Two positive claims are stronger than the corresponding tables. Several post-trained hosts decline on MMBench, MMStar, or Visual-CoT: ZwZ-8B drops by 3.13 points on MMStar, and ZwZ-4B drops by 1.85, 3.00, and 2.81 points on MMBench, MMStar, and Visual-CoT, respectively (Table 11). Table 4, in the main text, reports only the four foundation hosts. Learned fusion adds little accuracy. On Qwen3-VL-8B, replacing the learned bridge with an identity mapping lowers the five-dataset macro average only from 67.86 to 67.47 (Table 7). With the selected regions held fixed, internal fusion and the host's native multi-image interface obtain the same pooled accuracy, 66.57 (Table 13).

Questions for authors
  1. Under a matched visual-token budget, how does EviViT compare with V*, Mini-o3, FOCUS, and Divide, Conquer and Combine (Wang et al., 2025)? Does an attachment of this design transfer to a host whose vision encoder is neither Qwen3-VL nor Qwen3.5?

  2. On ZwZ and Vision-OPD, EviViT reduces accuracy on MMBench or MMStar, in contrast to the Qwen foundation hosts (Table 11). Why do these hosts differ from the Qwen hosts, and could this comparison be reported in the main text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.