- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CounterGround, a paired counterfactual benchmark for visual grounding. Each pair shares the same image, object category, and candidate set, while a minimal spatial relation edit changes the referred target instance. The benchmark contains 5,635 pairs from DIOR and COCO. The authors evaluate several MLLMs and find that models can respond consistently to relation changes and associate their predictions with the intended instances while still failing the bounding-box IoU criterion. They further conduct controlled experiments to examine the differences between target selection, center localization, and box extent prediction. Based on these observations, they propose a repair pipeline that combines joint geometric refinement with local visual-language features. Experiments show improvements in paired grounding accuracy on CounterGround and additional gains on RefCOCO and VRSBench under dataset-specific training protocols.
Critical strengths
-
An interesting perspective on visual grounding evaluation. I like the idea of evaluating grounding through paired target switches rather than treating each image-query pair independently. The controlled setup makes it possible to examine how models respond when a small change in the referring expression requires a different target instance. The distinction between directional consistency, instance association, and box accuracy is useful for understanding different types of grounding failures. I also appreciate that the authors evaluate multiple MLLMs on both remote-sensing and natural images.
-
The paper connects its analysis to a practical improvement. Instead of stopping at benchmark construction, the authors investigate possible sources of localization errors and propose a corresponding refinement method. The experiments are reasonably comprehensive, including component ablations, held-out evaluation, two MLLM backbones, and additional grounding benchmarks. The improvement in Pair Accuracy from 48.65% to 61.33% on CounterGround-DIOR validation is encouraging.
Critical weaknesses
-
The current metrics may overestimate the model's ability to identify the correct instance. My main concern is how the paper distinguishes correct instance association from coarse spatial responses. Direction Accuracy only checks whether the predicted center moves in the expected direction, while Instance Association Accuracy assigns each predicted center to its nearest same-class GT candidate. These metrics are useful, but they may not be strong enough to establish that the model has actually identified the intended object. For example, with a leftmost/rightmost pair, a model could place its predictions toward opposite sides of the image and potentially satisfy both criteria without accurately locating either object. This matters because the separation between instance association and box localization is the central finding of the paper. I would find the conclusion more convincing if the authors also reported stricter measures of instance localization, such as whether the predicted center falls inside the target box or its distance to the GT center normalized by object size. The benchmark is also restricted to two types of extremal spatial relations, which limits how broadly the findings can be interpreted.
-
The diagnosis and repair experiments do not fully isolate the sources of improvement. The GT-candidate selection experiment is informative, but it provides the model with exact candidate boxes, removing much of the difficulty involved in object detection and box prediction. The authors acknowledge this limitation, but the large gap between GT-candidate selection and free-form grounding does not by itself establish which component of ordinary grounding is responsible for the failures. I also have some reservations about the repair experiments. The proposed pipeline introduces additional trainable geometry and local-feature refinement modules, while the main baseline uses the original MLLM predictions. Although the paper includes center-only and direct-crop controls, a comparison with a conventional learned box refiner trained using the same data and local features would make the contribution clearer. One interesting detail in Table 3 is that joint geometric refinement improves Pair Accuracy from 48.65% to 56.06%, while mean IoU slightly decreases from 55.49 to 55.30. This suggests that the improvement in threshold-based accuracy does not necessarily translate into better overall box overlap. I would like to see further analysis of this behavior.
Questions for authors
Q1. Could the authors evaluate instance association using a stricter criterion than nearest-GT assignment? For example, how often does the predicted center actually fall inside the intended target box, and how does the center localization error vary with object size? I think this would help establish whether the reported association/localization gap reflects correct target identification or simply coarse spatial agreement.
Q2. How much of the observed behavior can be explained by simple spatial heuristics? Since the current benchmark only includes leftmost/rightmost and topmost/bottommost switches, it would be useful to examine performance across different numbers and spatial densities of same-class objects. The GroundingDINO + resolver baseline is helpful, but a more detailed analysis of candidate detection quality and a simple spatial-ranking strategy could further clarify the difficulty of the benchmark.
Q3. Could the authors compare the repair pipeline with a standard learned bounding-box refinement baseline under matched training data and feature inputs? Also, why does joint geometry refinement improve Pair Accuracy while slightly reducing mean IoU? Reporting center errors, box-size errors, and confidence intervals for the held-out repair results would help clarify where the gains come from.
These points, particularly the validity of the instance association measure and the controlled evaluation of the repair module, would influence my overall assessment.
4 comments
To Q3:
We added a conventional proposal-relative residual MLP under closely matched conditions. It receives the same 578 deployable inputs as our joint geometry + local evidence refiner, uses the same DIOR training split, validation roster, input preprocessing, box-level supervision, optimization budget, three seeds, and nearly identical trainable capacity (90,301 vs. 89,995 parameters). The validation split is not used for training, and all compared refiners use the same fixed-final checkpoint policy.
| Method | Pair [email protected] | mIoU (%) |
|---|---|---|
| Free-form grounding | 49.09% | 55.45 |
| Joint geometry refinement | 57.15% | 56.07 |
| Matched residual refiner | 59.65% | 57.03 |
| Joint geometry + local evidence | 61.17% | 58.44 |
The conventional residual refiner therefore explains most of the repair gain. Joint geometry + local evidence provides an additional +1.52 pp Pair Accuracy over this matched baseline, with a paired-bootstrap 95% CI of [+0.75, +2.28] using 10,000 complete-pair resamples. Its mean-IoU improvement over the matched residual baseline is also positive (+0.0140, 95% CI [+0.0122, +0.0159]). We accordingly narrow the claim: repairability is not unique to our architecture; joint geometry + local evidence provides a smaller but statistically supported incremental improvement. We will also disclose that two architecture-specific regularizers in its structured decoder have no direct counterpart in the conventional MLP, so the complete objectives are not literally identical.
The fresh rerun does not reproduce the slight mean-IoU decrease observed in the original table. Joint geometry refinement increases Pair Accuracy from 49.09% to 57.15% (+8.06 pp, 95% CI [+6.05, +10.14]), while mean IoU changes from 0.5545 to 0.5607 (+0.0062, 95% CI [−0.0043, +0.0170]). Thus, the previous slight decrease is not a stable effect. A pair-level transition analysis further shows that, averaged over three seeds, 17.42% of pairs change from failure to success, whereas 9.36% change from success to failure. This confirms that Pair Accuracy primarily reflects net crossings of the IoU = 0.5 threshold and therefore need not change proportionally with mean IoU.
We further analyzed localization geometry as requested. The target-size-normalized center error decreases from 0.962 for free-form prediction to approximately 0.558 with joint geometry refinement, 0.562 with the matched residual refiner, and 0.555 with joint geometry + local evidence. Absolute log-area error, used as a box-size error measure, decreases from 0.663 to 0.451, 0.401, and 0.397, respectively. These results indicate that most center correction is already achieved by geometric refinement, while learned proposal-relative refinement substantially improves box extent; joint geometry + local evidence provides a further, smaller improvement in localization quality.
Overall, conventional proposal-relative regression accounts for most of the repair effect, while joint geometry + local evidence provides a smaller but statistically supported additional gain. We will revise the manuscript to report the matched baseline, uncertainty intervals, center/size errors, and this narrower interpretation.
Note that the experiments in the original submission were conducted on two NVIDIA RTX 5090 GPUs, while the additional rebuttal experiments were conducted on two NVIDIA RTX 4090 GPUs.
Appreciate the authors for the extensive rebuttal, which addresses most of my concerns, I have accordingly raised my overall score to weak accept.
Thank you for your feedback and for supporting our previous rebuttal.
Thank you for the detailed and constructive feedback.
To Q1:
We agree that nearest-GT assignment is permissive and should not be interpreted as evidence of semantic instance identification. We therefore added a stricter Center-In-Target (CIT) criterion, requiring the predicted center to fall inside the intended target box, together with target-size-normalized center-error analysis. We will also rename the original metric as nearest-candidate geometric association to emphasize that it measures coarse geometric support rather than semantic recognition.
We report pair-level rates below, requiring each criterion to hold for both members. Headline pair-level metrics use all benchmark pairs, with invalid predictions counted as failures.
CIT remains above Pair Accuracy by 12.33 pp [11.38, 13.29] and 12.90 pp [11.92, 13.90] on DIOR, and by 8.22 pp [6.56, 9.88] and 7.53 pp [5.97, 9.10] on COCO (10,000 paired bootstrap resamples over complete pairs). Object scale also matters: for DIOR/Qwen3.5-2B, sample-level CIT rises from 46.15% on small targets to 83.12% on large targets, while normalized center error decreases from 5.14 to 0.32.
We will therefore narrow the original interpretation: nearest-candidate association is not evidence of correct instance recognition. However, the stricter CIT analysis still shows that predicted centers fall inside the intended target more often than the model produces an accurate box, so the gap cannot be explained solely by coarse left/right or top/bottom placement.
To Q2:
We further analyzed the GroundingDINO + deterministic extremal spatial resolver baseline. On COCO, it achieves 42.95% Pair Accuracy, versus 44.52% and 46.48% for Qwen3.5-2B and Qwen3-VL-2B; the differences are −1.57 pp [−4.60, +1.47] and −3.52 pp [−6.56, −0.39]. Thus, detection followed by deterministic extremal ranking constitutes a strong control on COCO and reaches performance close to the evaluated MLLMs. On DIOR, however, the same pipeline reaches only 15.65%, versus 37.37% and 36.27%; the gaps are −21.72 pp [−23.09, −20.35] and −20.62 pp [−22.00, −19.23].
Detection quality helps explain this cross-domain difference: at IoU ≥ 0.5, intended-target recall is 68.30% on COCO but 30.29% on DIOR, and detection failures account for 27.64% versus 62.50% of members. We also stratified MLLM performance by same-class candidate count and spatial density. On COCO/Qwen3.5-2B, Pair Accuracy falls from 49.39% with two candidates to 27.55% with four or more, and from 50.44% in sparse scenes to 35.48% in dense scenes; Qwen3-VL shows the same trend. DIOR does not show the same monotonic pattern, so we treat these results as descriptive within-dataset analyses rather than evidence of a causal effect.
We will therefore clarify that CounterGround evaluates controlled extremal target switching, not general spatial reasoning. The detector-plus-extremal-resolver baseline is competitive on COCO when detection is reliable, whereas it remains substantially below the evaluated MLLMs on DIOR, where candidate detection itself is a major bottleneck.