- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LazyMCoT, a training-free visual grounding framework designed to improve fine-grained visual understanding while reducing unnecessary inference costs. The method consists of two components: Adaptive Routing, which uses first-token statistics and conformal calibration to decide whether additional grounding is needed, and Collaborative Grounding, which combines VLM attention with an external visual expert through a two-stage refinement process. Experiments on V* Bench, HR-Bench, and TreeBench across three VLM backbones show improvements in overall accuracy and average inference latency.
Critical strengths
-
The paper identifies a relevant limitation of existing training-free grounding methods: applying visual manipulation to every sample can introduce unnecessary computation and sometimes hurt predictions that the base VLM already gets right.
-
The proposed framework combines lightweight uncertainty-based routing with attention-guided visual grounding. Using first-token statistics to decide whether additional visual processing is needed is a practical design choice, while combining VLM attention with an external visual expert offers complementary sources of localization information.
-
The paper evaluates the method on three VLM backbones and several high-resolution visual understanding benchmarks. The experiments include component ablations, threshold analysis, latency comparisons, and qualitative examples.
Critical weaknesses
-
One of the main motivations of the paper is that indiscriminate grounding can hurt performance on easy samples. However, Table 3 shows that applying Collaborative Grounding to all samples already substantially improves accuracy over the base VLM, with results close to those of the full method. In comparison, the additional accuracy gains from Adaptive Routing are relatively modest, suggesting that its main benefit may lie in reducing inference latency rather than improving accuracy. It would be helpful if the authors could further analyze how often Adaptive Routing prevents grounding from turning originally correct predictions into incorrect ones, thereby providing stronger evidence for the paper's central motivation.
-
The paper uses the predicted error probability of the base VLM to decide whether to trigger Collaborative Grounding. However, a wrong prediction does not necessarily mean that additional grounding will help. Some errors may result from insufficient visual information, while others may stem from the model's limited reasoning ability, which grounding may not resolve. This raises a question about why the authors choose to predict whether the base VLM is wrong, rather than whether grounding is likely to improve the answer. It would be helpful to clarify this design choice and report how many of the samples routed to grounding are actually corrected. A comparison with a routing strategy that directly predicts the benefit of grounding would also be valuable.
-
The use of conformal calibration to control the miss rate of difficult samples is reasonable, but I have a few questions about its theoretical guarantees. First, the routing threshold is determined by the empirical quantile of the must-recall samples. It is unclear whether this provides a finite-sample guarantee on unseen data. Second, the authors use out-of-fold scores for calibration, while the final router may differ from the models used to produce these scores. It would be helpful to clarify whether this affects the validity of the guarantee.
Questions for authors
Please refer to the Critical Weaknesses section.
Kindly Reminder
Dear Reviewer,
Thank you once again for your thoughtful and valuable comments! As the deadline for the rebuttal discussion approaches, we would greatly appreciate knowing if our responses have adequately addressed your inquiries. Should you require any further clarifications or have any additional concerns, please do not hesitate to reach out to us.
Your feedback is immensely valuable to us.
Warm regards, Authors.