acceptodds
Under review as a conference paper at ICLR 2027

Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

Abstract

While Multimodal Large Language Models (MLLMs) excel in cross-modal reasoning, they often struggle to perceive fine-grained details in complex high-resolution images. Recent training-free methods address this through image scaling and localized cropping. However, applying these manipulations indiscriminately introduces computational redundancy for simple queries and can degrade accuracy by truncating essential global context or introducing irrelevant background noise. To this end, we propose **LazyMCoT**, a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty. The framework features an ***Adaptive Routing*** mechanism that evaluates predictive uncertainty using first-token statistics from a single forward pass. This efficiently bypasses confident cases while ensuring the recall of difficult samples via conformal calibration. For these challenging cases, a ***Collaborative Grounding*** module integrates the inherent cross-modal attention of the model with an external visual expert through a two-stage refinement process. This refinement process generates a precise localized display to recover small or occluded targets. Extensive experiments across diverse benchmarks demonstrate that LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Grounding, Vision-Language Model, Multi-modal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9fAAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LazyMCoT, a training-free visual grounding framework designed to improve fine-grained visual understanding while reducing unnecessary inference costs. The method consists of two components: Adaptive Routing, which uses first-token statistics and conformal calibration to decide whether additional grounding is needed, and Collaborative Grounding, which combines VLM attention with an external visual expert through a two-stage refinement process. Experiments on V* Bench, HR-Bench, and TreeBench across three VLM backbones show improvements in overall accuracy and average inference latency.

Critical strengths
  1. The paper identifies a relevant limitation of existing training-free grounding methods: applying visual manipulation to every sample can introduce unnecessary computation and sometimes hurt predictions that the base VLM already gets right.

  2. The proposed framework combines lightweight uncertainty-based routing with attention-guided visual grounding. Using first-token statistics to decide whether additional visual processing is needed is a practical design choice, while combining VLM attention with an external visual expert offers complementary sources of localization information.

  3. The paper evaluates the method on three VLM backbones and several high-resolution visual understanding benchmarks. The experiments include component ablations, threshold analysis, latency comparisons, and qualitative examples.

Critical weaknesses
  1. One of the main motivations of the paper is that indiscriminate grounding can hurt performance on easy samples. However, Table 3 shows that applying Collaborative Grounding to all samples already substantially improves accuracy over the base VLM, with results close to those of the full method. In comparison, the additional accuracy gains from Adaptive Routing are relatively modest, suggesting that its main benefit may lie in reducing inference latency rather than improving accuracy. It would be helpful if the authors could further analyze how often Adaptive Routing prevents grounding from turning originally correct predictions into incorrect ones, thereby providing stronger evidence for the paper's central motivation.

  2. The paper uses the predicted error probability of the base VLM to decide whether to trigger Collaborative Grounding. However, a wrong prediction does not necessarily mean that additional grounding will help. Some errors may result from insufficient visual information, while others may stem from the model's limited reasoning ability, which grounding may not resolve. This raises a question about why the authors choose to predict whether the base VLM is wrong, rather than whether grounding is likely to improve the answer. It would be helpful to clarify this design choice and report how many of the samples routed to grounding are actually corrected. A comparison with a routing strategy that directly predicts the benefit of grounding would also be valuable.

  3. The use of conformal calibration to control the miss rate of difficult samples is reasonable, but I have a few questions about its theoretical guarantees. First, the routing threshold is determined by the empirical quantile of the must-recall samples. It is unclear whether this provides a finite-sample guarantee on unseen data. Second, the authors use out-of-fold scores for calibration, while the final router may differ from the models used to produce these scores. It would be helpful to clarify whether this affects the validity of the guarantee.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_QeCeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LazyMCoT, a training-free method for high-resolution image QA. The motivation is that existing crop/zoom methods process every sample, which wastes compute and can hurt easy samples. A router uses two first-token statistics (top option probability and option vs. non-option logit gap) with a GBDT trained on external data, and a conformal-style threshold decides whether to answer directly. Hard samples go through Collaborative Grounding: the VLM decomposes the question into entities, boxes come from VLM attention and SAM3, uncovered attention boxes are cropped and re-queried with SAM3, and the result is rendered as a localized panel display for a second VLM query. On V*, HR-Bench and TreeBench with three 7–8B VLMs, the method improves over the base models and is slightly better than HiDe.

Critical strengths
  1. The motivation is clear and useful: not every sample needs cropping or zooming, and Fig. 3 shows well how forced grounding can hurt an easy sample.

  2. The router is trained on external data with deduplication and leakage checks, and code is released.

Critical weaknesses
  1. The router, which is the core idea of the paper, adds very little. At the chosen α=0, the threshold is the minimum must-recall score and only 24.7% of samples are skipped, far from the 67% that the motivation suggests can be solved directly. Adding the router changes results by +0.6 on V* (about one question out of 191), +0.2 on HR-4K and +0.5 on HR-8K, from single runs. Table 5 shows that when the router skips more, accuracy drops. The "conformal guarantee" also does not hold here, since calibration data is chosen to differ from the test distribution and α=0 simply takes the minimum.

  2. The pipeline is heavily hand-designed. It stacks a GBDT on two hand-picked features, a long rule-based entity decomposition prompt, an external SAM3 service, per-VLM choices of attention layers, and many thresholds (σ, τ, several IoU and area cut-offs, expansion ratio, box count limits). This makes it hard to know which parts matter, hard to transfer to new models or open-ended questions (the router depends on option letters), and unlikely to scale. Recent work moves toward models that learn when and where to look end to end, and this paper does not show why a hand-built pipeline is the better direction.

Questions for authors

In Table 3, "w/o AR + CG" gives exactly the same numbers as the reimplemented HiDe on all four benchmarks (90.0 / 77.4 / 74.9 / 40.0), although the text says this setting uses SAM3 and Stage 2 on every sample while HiDe does not. Is this a copy error?

1 comment
AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer,

Thank you once again for your thoughtful and valuable comments! As the deadline for the rebuttal discussion approaches, we would greatly appreciate knowing if our responses have adequately addressed your inquiries. Should you require any further clarifications or have any additional concerns, please do not hesitate to reach out to us.

Your feedback is immensely valuable to us.

Warm regards, Authors.

Reviewer_EtJAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LazyMCoT, an adaptive visual grounding framework for MLLMs. The key motivation is that indiscriminately applying visual grounding operations, such as cropping and zooming, introduces unnecessary computational overhead and may even degrade performance on easy samples that can already be solved by the original model. To address this issue, it uses probability statistics from the first answer token and a GBDT router to determine whether to answer directly (AdaptiveRouting). For difficult examples, it combines VLM attention with SAM3 in a two-stage grounding process to construct a panel of local image regions, which is then passed to the VLM for answering (CollaborativeGrounding). Experiments across several visual reasoning benchmarks demonstrate improvements in accuracy and inference efficiency over compared baselines.

Critical strengths
  • The motivation is empricaly studied in the paper, showing that additional cropping increases computation and can also disrupt context that was already sufficient for answering. The problem setting is relevant.

  • Results across models are positive. The paper evaluates multiple VLM backbones and visual reasoning benchmarks. The reported improvements in accuracy and inference latency suggest that selectively allocating visual grounding computation can be beneficial in practice.

Critical weaknesses
  • Significant overlap with closely related prior work. The paper does not discuss the closely related work: LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models (originally versioned as AdaFocus, arXiv:2603.00171, February 2026). Specifically, LookWise/AdaFocus identifies the computational redundancy and potential accuracy degradation caused by indiscriminate visual cropping and proposes a confidence-based mechanism to determine when additional visual processing is necessary. It then introduces a training-free adaptive visual reasoning framework that selectively performs additional visual operations. These ideas substantially overlap with the central motivation and high-level formulation of LazyMCoT. Although the two methods different in specifiv implemention, the paper should clearly distinguish these incremental contributions from the broader adaptive grounding paradigm already established in prior work and direct compare with LookWise/AdaFocus.

  • The paper trains its GBDT router to distinguish originally correct and incorrect predictions. However, these categories do not necessarily correspond to whether additional visual grounding is beneficial. Although the conformal calibration procedure focuses on a must-recall subset, it is unclear whether the learned routing score is optimally aligned with the actual benefit of invoking the grounding module.

Questions for authors
  1. For question pairs that need different image regions, how often do early and deep attention retain the correct region for each question? Can you separate shared useful tokens from tokens that cause the early selector to miss the new evidence?

  2. How many answer failures come from Stage 1 removing needed evidence, rather than Stage 2 choosing poorly? Does the shared 33% prefix retain evidence for all questions about an image, especially small-text and small-object questions?

  3. Please also clarify the fidelity calculation in Table 2. For Qwen3-VL-8B FasterVLM, all eight listed scores improve from the 5% to the 10% setting, but average fidelity falls from 96.3% to 91.4%. Which entries need correction?

1 comment
AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer,

Thank you once again for your thoughtful and valuable comments! As the deadline for the rebuttal discussion approaches, we would greatly appreciate knowing if our responses have adequately addressed your inquiries. Should you require any further clarifications or have any additional concerns, please do not hesitate to reach out to us.

Your feedback is immensely valuable to us.

Warm regards, Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.