- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Motivated by everyday household questions, this paper studies visual question answering in cluttered tabletop scenes where the objects, spatial relations, or labels needed to answer a question may be occluded. The authors introduce MG-VQA-Bench, with 600 questions across COUNT, FIND, BENEATH, and COMPARE, each reviewed by two annotators. They also build MG-VQA-Sim, a physics-based environment with perception, segmentation, grasping, and pushing tools. Eight VLMs are evaluated in Direct, Perception, and Manipulation modes, with average success rates of 36.2%, 37.6%, and 56.8%, respectively. GPT 6 Astra reaches 83.0% in Manipulation mode (Table 1, p. 6). The paper examines search and recovery through prompt ablations and action statistics, and introduces the 40-question MG-VQA-Hard benchmark, where five models score between 17.5% and 27.5% (Tables 2–5, pp. 7–9). These results support the benefit of manipulation for answering questions in this tabletop simulation with its fixed tool set, and show differences in how models gather evidence and handle difficult cases.
Critical strengths
-
The benchmark provides a useful way to evaluate evidence gathering through manipulation. The four tasks cover finding hidden objects, counting all instances, revealing spatial relations, and identifying occluded labels. The authors combine occlusion filtering and review by two annotators with grasping and pushing actions whose physical consequences persist across turns (§3.1–3.3, pp. 3–5; Appendix C, pp. 16–18). This setup lets researchers examine how VLM agents respond to insufficient evidence and failed actions, which is relevant to embodied question answering and VLM planning.
-
The comparisons and behavioral analyses are informative. The comparison of eight models across three modes on the same 600 questions shows that all models achieve higher average success rates in the Manipulation mode (Table 1). Figure 3 further examines the relationship between final answer outcomes and use of the Manip tool. Together, these analyses provide a well-supported basis for distinguishing models’ willingness to use manipulation tools from their ability to effectively use the evidence obtained through manipulation.
Critical weaknesses
-
Some evaluation rules are unclear or inconsistent. In Figure 8 (p. 19), the system prompt allows perception and answering after the 10-action budget is exhausted, but the caption says that exhausting the budget terminates the episode and counts it as incorrect. FIND also requires a target point for positive answers, yet the paper does not specify how that point is judged correct (§3.1; Appendix C.4; Figure 8). These gaps make the evaluation harder to reproduce and affect the interpretation of failures. In particular, an unsuccessful positive FIND answer may reflect a localization error rather than a failure to find the target. The text needs clarification; the available evidence does not establish that the reported scores are wrong.
-
The stability of the smaller comparisons is unclear. I am less confident about the prompt-ablation differences and the rankings on MG-VQA-Hard than about the large gains from allowing manipulation. Qwen 3.8 Flash Next scores 54.0% on the main benchmark, below Astra's 83.0%, but scores 27.5% versus Astra's 25.0% on Hard. Since Hard has only 40 questions, this lead amounts to one additional correct answer and may reflect the particular questions in this small set. Astra's Search drop from 83.0% to 80.5% also amounts to a net difference of 15 answers out of 600. The paper does not report repeated-run variability or uncertainty estimates for these comparisons. I would like to know whether the smaller trends persist across runs and balanced resamples of the existing questions, and how sensitive they are to model-output randomness and evaluation sample size.
Questions for authors
-
How much of the FIND present/absent gap comes from the scoring requirements? Each group has 75 questions. A positive answer requires both answering Yes and providing a correct target point, whereas a negative answer requires only answering No. This asymmetry may contribute to the higher success rate on absent-target cases in Figure 3. Could you specify the point-scoring rule and separate presence-decision errors, localization errors, and episodes that terminate without an answer? This would help assess how much of the gap is due to insufficient search.
-
Why does Astra’s performance decrease with Search? Under the Search ablation, Astra’s average success rate drops from 83.0% to 80.5%, even though manipulation occurs in 584 episodes in both settings (§4.1, p. 7). Could the authors explain this decrease? In particular, could the observed 2.5-percentage-point drop reflect sampling variability due to the limited evaluation size, rather than a systematic negative effect of the Search prompt?
-
Why does Opus 4.8 perform worse with Search-persistent than with Search? Table 2 shows that Opus 4.8’s average success rate improves from 41.5% to 44.0% with Search, but drops back to 41.5% with Search-persistent. Meanwhile, the number of episodes involving manipulation increases from 293 to 305. Could the authors explain why performance declines despite manipulation being used in more episodes?
4 comments
Evaluation clarifications, FIND errors, and repeated-run results (2/2)
Response to Q3:
We repeated the Search and Search-persistent evaluations across three runs -
| Model | Mode | Count | Find | Beneath | Compare | Average | Acted / 600 | Mean Manip./ep |
|---|---|---|---|---|---|---|---|---|
| Opus 4.8 | Baseline | 26.0 ± 0.7 | 48.2 ± 1.0 | 50.9 ± 0.8 | 44.0 ± 2.0 | 42.2 ± 0.8 | 228.0 ± 3.6 | 1.1 ± 0.0 |
| Opus 4.8 | Search | 27.8 ± 2.0 | 48.7 ± 0.7 | 51.3 ± 3.5 | 45.6 ± 2.3 | 43.3 ± 0.9 | 273.0 ± 17.4 | 1.2 ± 0.1 |
| Opus 4.8 | Search-persistent | 28.9 ± 3.2 | 50.0 ± 0.7 | 49.1 ± 3.9 | 44.9 ± 6.1 | 43.2 ± 1.6 | 291.3 ± 13.1 | 1.3 ± 0.0 |
Opus 4.8 shows a narrower improvement in average success rate from baseline to search. The general finding still remains: the ‘Search’ prompt encourages the model to manipulate in more tasks. Opus achieves 43.3 ± 0.9% with Search and 43.2 ± 1.6% with Search-persistent. The repeated evaluations show a smaller increase from baseline to search and a smaller decrease in average performance with Search and Search-persistent, suggesting that the original differences reflect run-to-run variability. We will revise the discussion in the final version to emphasize that repeating the guidance provides no demonstrated additional benefit over Search in these evaluations, rather than claim that it reduces performance.
We hope the clarified protocol and additional analyses address your questions. We welcome further discussion.
Following up on our rebuttal
We appreciate your careful review and positive assessment. As the discussion period comes to a close, we wanted to follow up on whether our responses addressed your questions.
To recap: we have (i) clarified the Figure 8 caption: exhausting the budget does not itself end the episode, while five consecutive perception-only calls do and are scored as incorrect; (ii) specified the FIND point-scoring rule (the point must lie inside the ground-truth target mask in the final frame) and broken down FIND failures, showing that wrong presence decisions outnumber localization errors for every model; (iii) added 5-seed results on MG-VQA-Hard and will revise the ranking discussion accordingly (Sec. 4.3, Table 5); and (iv) repeated the Search and Search-persistent ablations over 3 seeds, which show that the Astra and Opus differences are within run-to-run variability. We will revise those claims in the final version (Sec. 4.3).
If our responses have resolved your concerns, we would be grateful if you would consider reflecting this in your final rating. We are glad to address anything further in the time left.
Thank you to the authors for the thoughtful clarifications and extensive additional experiments. I have carefully read the rebuttal and the other reviewers' comments, and I believe most of my concerns have been addressed.
In particular, the authors have provided additional experimental results for Table 2: Ablations conducted on MG-VQA-Bench (600 questions), offering more evidence for evaluating the findings. However, these results also seem to suggest that the ablation study does not yield sufficiently consistent conclusions, as the trends observed across Opus 4.8, Kimi K3, and GPT 6 Astra are not aligned. My interpretation is that these strong models may be relatively insensitive to additional cues, given their already high baseline performance. I believe this observation deserves a more thorough discussion in the paper.
Overall, I am still not fully convinced by the responses to Q2 and Q3, although I do not consider these remaining concerns to be major obstacles. In my view, the motivation and design of this benchmark are well aligned with the future direction of embodied AI research, which I consider a meaningful contribution.
Therefore, I have decided to maintain my rating of Weak Accept.
Evaluation clarifications, FIND errors, and repeated-run results (1/2)
Thank you for your careful review and specific questions about scoring rules and result stability.
Correction in the caption: We thank the reviewer for pointing this out. The system prompt in Figure 8 correctly describes budget exhaustion: after 10 manipulation attempts, models may still invoke perception tools or answer, and exhausting the manipulation budget does not itself count as failure. Five consecutive perception-only calls trigger automatic termination and are scored as incorrect. We will correct the caption in our final version.
Response to W1/Q1:
For FIND, a target point is judged correct when the returned point lies within the ground truth target mask from the simulator in the final top-down frame, and yes, a positive answer requires both “Yes” and a correct target point to be successful. We will clarify this in our final version.
We provide a detailed breakdown of the failure cases for FIND. We categorize failures into three types: (1) Incorrect choice (VLM returned yes when ground truth is no and vice versa); (2) localization errors (VLM returned yes, ground truth is yes but 2D point does not lie on the privileged simulator mask for that termination scene step); (3) Episode termination failures (VLM used 5 perception tool calls in a row which led to automatic termination and task failure) as shown below.
The results show that incorrect choice dominates localization errors for all models.
Response to W2:
Thank you for your suggestion. We further report repeated-run variability and uncertainty estimates on MG-VQA-Hard by measuring the mean and variance on the models’ performances across 5 seeds as shown below.
Consistent with our original findings, our updated results show that all models underperform on MG-VQA-Hard compared to MG-VQA-Bench. Moreover, table-top scenes that appear uncluttered but hide hidden objects underneath the clutter expose the bias in all models to answer questions early without investigating and verifying if any evidence is hidden. We do now see that Gemini 3.1 Pro leads over Qwen 3.8 Flash Next. We will include this in our final version.
Response to Q2:
We further look into the repeated-run variability and uncertainty estimates in the Search ablation by running it on MG-VQA-Bench with three different seeds -
The above results show that the performance for GPT 6 Astra is statistically similar with and without the Search prompt. Additional guidance therefore produces little change in aggregate interaction frequency. We thank the reviewer for pointing this out and we will update this in our final version.