acceptodds
Under review as a conference paper at ICLR 2027

Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement

Abstract

Vision–language models can identify the correct referent while returning an imprecise bounding box. We study whether a frozen direct-answer model can use its own prediction to allocate and safely exploit a localized re-observation without accessing target annotations at inference. Label-free precision refinement (LFPR) routes predicted-small regions to a higher-resolution pass, re-grounds the expression inside a context crop, admits a candidate only under fixed geometric guards, and returns a fixed coordinate-wise midpoint. We report results across three evidence tiers. On 31,921 Ref-L4 expressions – retrospective evidence, since the published test distribution informed method development – LFPR raises our primary endpoint, mAcc(0.5:0.95), from 72.947% to 76.013% ([email protected] 88.531% to 89.725%, [email protected] 55.788% to 61.142%). A frozen, no-retuning transfer to 30,969 RefCOCO/RefCOCO+/RefCOCOg expressions – a later content-hash audit found substantial image overlap with Ref-L4 – improves every dataset individually at [email protected], mAcc, and mean IoU (mAcc +0.645, [email protected] +0.817, mean IoU +0.552 points pooled), while [email protected] is not distinguishable from zero on any dataset or pooled – a stage-wise audit shows routing alone gains +1.162 points at [email protected] while crop, guards, and fusion give back -1.192 points relative to that routed arm, so the two stages substantially offset rather than the operator having no strict-IoU effect. A prospective, image-disjoint Flickr30K Entities evaluation improves every endpoint under an MDETR-style merged-box protocol (mAcc +0.973, [email protected] +1.022) and more strongly under a historical single-box selection on the same images (mAcc +2.575, [email protected] +3.689). Applying the identical operator to two released grounding specialists improves every reported endpoint ([email protected] +1.569 for EGM-4B, +6.716 for EGM-8B) at roughly twice the inference latency, showing that the refinement composes with specialist training rather than replacing it. On the untuned transfer and prospective tiers, a genuine unguarded control – the same candidates with the guard removed – underperforms the incumbent, showing the guard is load-bearing. Together, these results show that referent selection and boundary precision are partially separable, and that different inference components move different, sometimes opposing, regions of the IoU curve – behavior a single accuracy threshold cannot reveal.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual grounding, referring expression comprehension, vision-language models, training-free methods, test-time refinement, bounding-box localization, IoU threshold evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BcjbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The work studies whether a frozen VLM can refine its own bounding box predictions for referring expression comprehension without access to ground-truth labels during inference. The authors propose Label-Free Precision Refinement, which routes small predicted regions to a higher-resolution pass, re-grounds inside a context crop under geometric guards, and fuses the results using a fixed midpoint. Experiments across multiple benchmarks show that LFPR improves overall performance such as mAcc and Acc @ 0.5, while demonstrating that different components affect different parts of the IoU curve.

Critical strengths
  1. LFPR operates entirely at inference time on a frozen model without requiring weight updates, auxiliary training datasets, or external object detectors, making it easy to integrate into existing vision-language model pipelines.

  2. The experiments setups are designed across multiple tiers, including retrospective studies, cross-dataset transfers without retuning, and prospective evaluations on Flickr30K, supported by detailed latency and resource audits.

Critical weaknesses
  1. While the method improves permissive metrics (e.g., Acc @ 0.5), its performance at strict IoU thresholds (e.g., Acc @ 0.9) is unstable and exhibits negative marginal feedback. Ablations reveal that subsequent cropping, guarding, and fusion stages frequently offset or reverse the strict-IoU gains achieved by initial resolution routing—especially on untuned benchmarks like Flickr30K.

  2. The pipeline heavily relies on hardcoded constants such as size-bucket thresholds, crop scale factors, geometric guard limits, and fusion weights rather than adaptive or learned mechanisms. Grid sweeps demonstrate minimal room for Pareto improvement, indicating a strong vulnerability to specific heuristic tuning.

  3. The paper suffers from a critically sparse reference list (less than 20 in main text) and an inadequate discussion of related work. It fails to adequately compare against or position itself alongside a broader spectrum of recent state-of-the-art test-time scaling, perception refinement, or zero-shot grounding paradigms, making the baseline comparisons weak and unconvincing for a top-tier venue.

  4. The manuscript suffers from abysmal layout organization and poor presentation structure. The core text and an excessively lengthy, disorganized appendix blur together into a tedious read, creating a strong suspicion of artificial page-padding rather than rigorous academic packaging.

The w3 and w4 are sufficient evidence already showing that I do not this manuscript is ready for ICLR.

Questions for authors

Given that the marginal effect of the crop, guard, and fusion stages frequently neutralizes or reverses the strict-IoU gains achieved by initial resolution routing (especially on non-tuned datasets like Flickr30K), what specific visual or geometric failure modes in the raw crop candidates make the geometric guards reject or degrade predictions that routing alone successfully improved? Furthermore, how do the authors justify the lack of comprehensive comparisons against state-of-the-art test-time adaptation baselines given the sparse literature review (references)?

Reviewer_hXYFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The method is simple and inference-only. It uses higher resolution for small predicted boxes, predicts again in a crop, and checks the new box with fixed geometric rules. If the new box passes, the final box is the midpoint of the two predictions.

On Ref-L4, mAcc improves from 72.9% to 76.0%, and [email protected] from 55.8% to 61.1%. On RefCOCO/+/g, mAcc improves by 0.65 percentage points, but [email protected] is almost unchanged. On the image-disjoint Flickr30K set, [email protected] and mAcc improve by 0.82 and 0.97 points under the merged-box protocol. The method also improves two grounding specialists, but takes about twice the inference time.

Critical strengths

The method is simple and inference-only. The ablations show the geometric checks matter, without the geometric checks, the cropped box is worse than the original prediction on the transfer set and on Flickr30k.

Critical weaknesses

The gain seems small. On Flickr30k, the only set not used in development, [email protected] is only +0.82 and mAcc +0.97, but latency is about 2×. The bigger [email protected] gain (+5.4) is on Ref-L4, which was used to build the method.

There are few visualizations. Most figures are curves and tables. I do not see qualitative examples.

The positive results mainly come from Qwen-based models. The GLM result is worse. This limits the evidence that the method works more generally.

Questions for authors

Please add some qualitative examples: inputes, baseline boxes, and improved boxes.

Reviewer_2siAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Label-Free Precision Refinement (LFPR), a method for improving the bounding box precision of Vision-Language Models (VLMs). The authors observe that while VLMs are generally strong enough to identify the correct object, they often return coarse bounding boxes that fail under strict evaluation metrics ([email protected]). LFPR addresses this issue through a two-step process: first, a size-bucket router forces "small" predicted boxes to be re-evaluated at a higher resolution; second, the model is prompted to re-ground the expression inside a localized context crop. To prevent this second pass from ruining a good initial guess, the crop candidate is only accepted if it passes strict geometric guardrails, after which it is fused with the original box via a coordinate midpoint.

Experiments validate LFPR across three datasets: Ref-L4, the RefCOCO family, and Flickr30K Entities. The authors also perform a stage-wise mechanistic audit to break down the pipeline's behavior. This ablation reveals a clear trade-off: resolution routing drives the improvements in strict boundary adherence, while the local re-grounding and fusion stages act defensively to protect lenient accuracy.

Critical strengths
  1. The authors take great care about the data governance using cyptographic content-hash audit, creating a valuable benchmark.
  2. The paper consists of detailed ablations to isolate the effect of different components of the pipeline.
  3. The results are interesting, especially the trade-off between strict vs lenient boundary adherence.
Critical weaknesses
  1. The "defensive protection" mechanism prevents the model from correcting a completely wrong first guess. Because the pipeline relies on restrictive geometric guards, it is tethered to the initial localization. If the first pass hallucinates or bounds the wrong object, the pipeline bends to accommodate this failure rather than allowing the model to freely correct itself.
  2. Although the authors limit resolution routing to small boxes to avoid a catastrophic compute penalty, the pipeline still doubles inference latency and token generation costs for a marginal ~0.8% gain in [email protected] on the RefCOCO family.
  3. The lack of high resolution images for larger boxes also makes the eval biased since the model would benefit from high resolution images regardless of the guessed box size.
  4. The geometric guards and heuristics need to be hardcoded for specific datasets and are not generalizable.
Questions for authors
  1. How does the LFPR technique compare against simply generating 2 to 3 independent bounding box predictions and averaging them in terms of performance and latency?
  2. Have you explored other ways of feeding global context - like adding a low-resolution full image along with cropped image or adding full image features in some way?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.