- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how the quantity of preselected input views affects vision-language models (VLMs) in zero-shot 3D visual grounding by introducing the Observe-Then-Ground (OTG) evaluation protocol, which fixes the downstream 3D instantiation pathway to isolate the impact of the visual evidence budget. Experimental results reveal that while GPT-5.6 and Gemini steadily improve their grounding accuracy as the evidence budget expands to Top-24, dense Qwen3-VL checkpoints experience performance degradation due to losing final 3D accuracy on already-visible targets. Ultimately, this paired-query analysis demonstrates that view preselection is not a model-independent computational shortcut, but rather an accuracy-relevant design choice that interacts deeply with the specific VLM and must be evaluated jointly.
Critical strengths
(1) The paper addresses a clear and practically relevant question: whether view preselection helps or hinders zero-shot 3D grounding across different VLMs. Studying the trade-off between reducing computation and preserving useful visual evidence offers valuable guidance for jointly choosing the grounding model and view budget.
(2) OTG provides a well-controlled evaluation protocol by fixing the VLM output format and downstream 2D-to-3D pipeline while varying only the supplied observations. The nested Top-k inputs derived from a shared ranking further reduce confounding factors, enabling a clearer analysis of view preselection.
(3) The experimental analysis goes beyond aggregate accuracy by distinguishing recovery of initially omitted targets from grounding changes on already-covered queries. Paired-query comparisons, direct 2D localization metrics, and scene-clustered bootstrap intervals provide stronger evidence for the model-dependent effects of view budgets.
Critical weaknesses
(1) The main study is restricted to 250 ScanRefer queries and a single PE-Core ranking. The alternative-selector experiment covers only GPT at Top-6, leaving it unclear whether the observed model-dependent responses persist across datasets, selectors, and sampling strategies. A broader selector × model × budget evaluation would strengthen the conclusions.
(2) Increasing the view budget simultaneously changes target coverage, distractor content, sequence length, and the positions of existing views. Although the paired analysis separates target recovery from changes on already-covered queries, it does not isolate why additional views harm some models. Target-preserving distractor controls and repeated order perturbations across backbones would provide stronger mechanistic insight.
(3) Each model–budget condition uses only one saved generation per query. The scene-clustered bootstrap intervals quantify uncertainty from scene resampling, but do not capture stochastic variation in model outputs. This is particularly relevant for small differences, such as GPT’s one additional correct prediction from Top-24 to Full. Repeated runs would help establish the stability of these findings.
(4) Lack of discussion of recent and highly relevant works [1]-[3]
[1] Visual programming for zero-shot open-vocabulary 3d visual grounding, 2024 CVPR
[2] Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding, 2026 EMNLP
[3] Solving zero-shot 3d visual grounding as constraint satisfaction problems, arXiv:2411.14594
Questions for authors
Please see the weakness.
12 comments
W2 (continued)
In contrast, presentation order has a substantial influence on performance: relevance ordering considerably reduces the performance degradation as the view budget increases to Top-24. For the 235 queries with fixed target coverage, the two orderings produce similar accuracy at Top-6, while the decreases in direct 2D accuracy from Top-6 to Top-24 are reduced from 7.2/15.0/11.5/17.9 pp under chronological ordering to just 0.4/6.7/4.3/4.0 pp under relevance ordering. Similarly, the decreases in final 3D accuracy are reduced from 11.1/10.5/7.5/3.7 pp to 4.5/4.3/3.1/2.6 pp (Table 4). The effect is particularly striking for the Unique queries, where the changes in direct 2D accuracy improve from −4.0/−16.2/−21.2/−26.8 pp under chronological ordering to +2.0/−3.0/−3.5/−3.0 pp under relevance ordering (Table 7).
By contrast, the more powerful VLMs, Gemini and GPT, remain substantially more robust to increasing view budgets, with changes in final 3D accuracy of no more than 1.3 pp from Top-6 to Top-24 across these comparisons. Together, these findings indicate that neither target coverage nor same-class distractors adequately explain the performance degradation of the smaller VLMs, while the substantial improvements achieved through relevance ordering highlight the importance of how additional views are presented.
The comparison of presentation orders also provides further insight into the source of the performance degradation observed in the smaller VLMs. Under both ordering methods, Qwen 4B, 8B, and 32B continue to select images containing the target almost as frequently at Top-24 as at Top-6, indicating that their loss in 2D accuracy arises primarily from less accurate bounding-box localisation rather than incorrect image selection (Table 4).
For all four Qwen models, bounding-box localisation is consistently more accurate when the selected image occupies one of the first three positions in the input sequence, regardless of presentation order or view budget. At Top-24, this occurs for 54–76% of selected images under relevance ordering, compared with only 18–28% under chronological ordering (Table 5).
In contrast, although Gemini and GPT also select images from later positions as the view budget increases, their bounding-box accuracy is considerably less sensitive to the position of the selected image. At Top-24, the difference in accuracy between selections from the first three positions and those from later positions is less than 8 percentage points (pp) for Gemini and GPT, compared with 12–30 pp for the Qwen models.
Since relevance ordering keeps the original six views in the first six positions and only appends the additional views, the performance degradation that persists under relevance ordering reflects the combined effects of increased sequence length and additional visual content. These findings demonstrate that additional views can degrade the performance of smaller VLMs even when target coverage is fixed and same-class distractors are absent, primarily through less accurate bounding-box localisation for Qwen 4B, 8B, and 32B. However, this degradation is substantially reduced when the additional views are appended after the more relevant views. In contrast, neither Gemini nor GPT shows a comparable gain from relevance ordering. These contrasting responses demonstrate that the same additional views affect the two groups of VLMs differently, reinforcing our central conclusion that view preselection should be evaluated in conjunction with the grounding VLM.
Table 3. Change from Top-6 to Top-24 for queries whose targets are visible at Top-6 and for queries whose targets are visible only at Top-24.
| Model | Targets visible at Top-6 (235 queries): 2D bbox [email protected] Top-24 minus Top-6 (pp) | Targets visible at Top-6 (235 queries): 3D bbox [email protected] Top-24 minus Top-6 (pp) | Targets visible only at Top-24 (15 queries): 3D bbox [email protected] at Top-24 (%) | All 250 queries: 3D bbox [email protected] Top-24 minus Top-6 (pp) | All 250 queries: of which from targets visible at Top-6 (pp) | All 250 queries: of which from targets visible only at Top-24 (pp) |
|---|---|---|---|---|---|---|
| Qwen3-VL-2B | −7.2 ± 0.9 | −11.1 ± 3.4 | 2.2 ± 3.8 | −10.3 ± 3.3 | −10.4 ± 3.2 | +0.1 ± 0.2 |
| Qwen3-VL-4B | −15.0 ± 0.9 | −10.5 ± 0.9 | 6.7 ± 0.0 | −9.5 ± 0.8 | −9.9 ± 0.8 | +0.4 ± 0.0 |
| Qwen3-VL-8B | −11.5 ± 0.4 | −7.5 ± 1.5 | 13.3 ± 0.0 | −6.3 ± 1.4 | −7.1 ± 1.4 | +0.8 ± 0.0 |
| Qwen3-VL-32B | −17.9 ± 4.5 | −3.7 ± 4.0 | 11.1 ± 3.8 | −2.8 ± 4.0 | −3.5 ± 3.8 | +0.7 ± 0.2 |
| GPT-5.6 | −0.9 ± 2.7 | −0.8 ± 2.1 | 28.0 ± 5.6 | +1.0 ± 1.9 | −0.7 ± 1.9 | +1.7 ± 0.3 |
| Gemini 3.8 Flash | +0.9 ± 1.3 | −0.4 ± 0.9 | 38.7 ± 3.0 | +1.9 ± 0.7 | −0.4 ± 0.8 | +2.3 ± 0.2 |
Targets visible at Top-6: the target appears in the Top-6 input, and therefore also in the Top-24 input (N=235). Targets visible only at Top-24: the target first enters the input at Top-24 (N=15); at Top-6 the accuracy on these queries is zero. [email protected]: the percentage of the queries of a group whose bbox has IoU ≥0.50 with the target. The last two columns split the change over all queries into the part that comes from each group: the change in the number of correct 3D bboxes in that group, as a percentage of all 250 queries. They add up to the change over all queries before rounding. Values are means ± sample SD over three runs for Qwen and five runs for GPT/Gemini; changes are computed within each run. View selector: CLIP. Views in chronological order.
Response to Reviewer Qgk8
We thank you for highlighting the factors that change simultaneously as the view budget increases. Your observations have helped guide the additional analyses presented below.
W1. Selector × model × budget and broader scope
You ask whether the model-dependent responses to increasing view budgets persist across different view selectors and suggest a broader evaluation spanning view selectors, models, and view budgets. To address this, we have extended our experiments to compare two view selection methods, CLIP (the PE-Core ranking used in the original submission) and SigLIP2, across all six VLMs at Top-3/6/12/24. Both selectors are evaluated using the same ScanRefer evaluation protocol and stride-30 candidate pool. In the original submission, SigLIP2 was evaluated only for GPT at Top-6 (Table 2).
The results demonstrate that the contrasting model-dependent trends persist across both view selection methods. As the view budget increases from Top-6 to Top-24, accuracy decreases for all four Qwen models (2B/4B/8B/32B), by 10.3/9.5/6.3/2.8 percentage points (pp) with CLIP and 14.0/8.8/8.4/4.4 pp with SigLIP2. In contrast, accuracy increases slightly for the more powerful VLMs, GPT and Gemini, by a mean of 1.0/1.9 pp with CLIP and by 1.2/1.2 pp with SigLIP2.
These findings confirm that the contrasting responses to increasing view budgets are consistent across the two view selectors evaluated, supporting our central conclusion that the influence of view budget on grounding accuracy depends primarily on the VLM rather than the choice of view selector.
Table 2. Selector × model × budget results with the SigLIP2 view selector: 3D bbox [email protected] (%).
3D bbox [email protected]: the percentage of all 250 queries whose final 3D bbox has IoU ≥0.50 with the target. The last column is Top-24 minus Top-6 in percentage points (pp). One run per condition. View selector: SigLIP2. Views in chronological order.
W2. Coverage, distractors, sequence length, and presentation order
You ask how we can identify why additional views degrade performance in some models, given that increasing the view budget simultaneously changes target coverage, distractor content, sequence length, and the positions of existing views. To distinguish these effects, we have extended our analysis to all six VLMs through three complementary comparisons, each controlling for different factors. Previously, only target coverage was examined separately, and the only presentation-order control involved a single shuffled ordering for GPT at Full.
The first comparison holds target coverage fixed by considering the 235 queries whose targets are already visible at Top-6 and therefore remain visible at Top-24. The second focuses on the 66 Unique queries, for which the target is the only object of its class in the scene, thereby excluding the possibility that additional views introduce same-class distractors. The third examines the influence of presentation order by comparing two arrangements of the same selected images: chronological order, which follows the temporal sequence of the scan, and relevance order, which ranks views by their similarity to the query. Importantly, relevance ordering preserves the original six views in the first six positions as additional views are introduced. Since both image ordering methods use exactly the same selected images at each view budget, distractor content and sequence length remain unchanged between them.
The results demonstrate that the performance degradation of the smaller Qwen models (2B/4B/8B/32B) persists even when target coverage is held fixed and when same-class distractors are absent. However, changing only the presentation order substantially reduces this degradation in direct 2D accuracy for all four Qwen models, with corresponding improvements in final 3D accuracy most pronounced for Qwen 2B, 4B, and 8B.
When target coverage is held fixed, increasing the view budget from Top-6 to Top-24 still reduces mean final 3D accuracy by 11.1/10.5/7.5/3.7 percentage points (pp) for Qwen 2B/4B/8B/32B, respectively (Table 3). For the Unique queries, where same-class distractors are also excluded, the corresponding decreases are 14.6/13.1/8.6/9.6 pp (Table 7), demonstrating that the degradation is at least as pronounced in the absence of such distractors.