acceptodds
Under review as a conference paper at ICLR 2027

MG-VQA: Manipulation Grounded Visual Question Answering with VLMs

Abstract

Vision-language models (VLMs) have shown promising spatial reasoning capabilities from static visual inputs, where the evidence needed to answer a question is available in the provided views. However, in cluttered environments, answer-relevant evidence may be occluded rather than absent: an object may lie beneath a pile, be covered by another object, or have identifying information facing away from the camera. Answering such questions requires physical interaction to reveal the hidden evidence. We formalize this setting as Manipulation-Grounded Visual Question Answering (MG-VQA), where an agent answers a question about an initially cluttered scene by using manipulation as an intermediate evidence-gathering operation. We introduce MG-VQA-Bench, comprising 600 human-verified questions across four spatial reasoning tasks, and evaluate it in a cluttered tabletop interactive environment. Eight VLMs are evaluated under three levels of environment access: Direct (image only), Perception (pointing, segmentation, and scene graphs), and Manipulation (perception, grasping, and pushing). Across all models, Direct and Perception achieve average success rates of 36.2% and 37.6%, respectively, only slightly above an image-blind chance baseline of 32.7%. In contrast, Manipulation raises average success to 56.8% (40.8-83.0% across models). Stronger tool-calling VLMs, GPT 6 Astra (83.0%) and Gemini 3.8 Flash (69.3%), search persistently and re-ground after unsuccessful interactions, while weaker models often answer without gathering sufficient evidence or stop prematurely after failed actions. Our results highlight the need for VLM agents that use manipulation for persistent, physically grounded evidence gathering and recovery.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Spatial reasoning, Visual Question Answering, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_43i9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by everyday household questions, this paper studies visual question answering in cluttered tabletop scenes where the objects, spatial relations, or labels needed to answer a question may be occluded. The authors introduce MG-VQA-Bench, with 600 questions across COUNT, FIND, BENEATH, and COMPARE, each reviewed by two annotators. They also build MG-VQA-Sim, a physics-based environment with perception, segmentation, grasping, and pushing tools. Eight VLMs are evaluated in Direct, Perception, and Manipulation modes, with average success rates of 36.2%, 37.6%, and 56.8%, respectively. GPT 6 Astra reaches 83.0% in Manipulation mode (Table 1, p. 6). The paper examines search and recovery through prompt ablations and action statistics, and introduces the 40-question MG-VQA-Hard benchmark, where five models score between 17.5% and 27.5% (Tables 2–5, pp. 7–9). These results support the benefit of manipulation for answering questions in this tabletop simulation with its fixed tool set, and show differences in how models gather evidence and handle difficult cases.

Critical strengths
  1. The benchmark provides a useful way to evaluate evidence gathering through manipulation. The four tasks cover finding hidden objects, counting all instances, revealing spatial relations, and identifying occluded labels. The authors combine occlusion filtering and review by two annotators with grasping and pushing actions whose physical consequences persist across turns (§3.1–3.3, pp. 3–5; Appendix C, pp. 16–18). This setup lets researchers examine how VLM agents respond to insufficient evidence and failed actions, which is relevant to embodied question answering and VLM planning.

  2. The comparisons and behavioral analyses are informative. The comparison of eight models across three modes on the same 600 questions shows that all models achieve higher average success rates in the Manipulation mode (Table 1). Figure 3 further examines the relationship between final answer outcomes and use of the Manip tool. Together, these analyses provide a well-supported basis for distinguishing models’ willingness to use manipulation tools from their ability to effectively use the evidence obtained through manipulation.

Critical weaknesses
  1. Some evaluation rules are unclear or inconsistent. In Figure 8 (p. 19), the system prompt allows perception and answering after the 10-action budget is exhausted, but the caption says that exhausting the budget terminates the episode and counts it as incorrect. FIND also requires a target point for positive answers, yet the paper does not specify how that point is judged correct (§3.1; Appendix C.4; Figure 8). These gaps make the evaluation harder to reproduce and affect the interpretation of failures. In particular, an unsuccessful positive FIND answer may reflect a localization error rather than a failure to find the target. The text needs clarification; the available evidence does not establish that the reported scores are wrong.

  2. The stability of the smaller comparisons is unclear. I am less confident about the prompt-ablation differences and the rankings on MG-VQA-Hard than about the large gains from allowing manipulation. Qwen 3.8 Flash Next scores 54.0% on the main benchmark, below Astra's 83.0%, but scores 27.5% versus Astra's 25.0% on Hard. Since Hard has only 40 questions, this lead amounts to one additional correct answer and may reflect the particular questions in this small set. Astra's Search drop from 83.0% to 80.5% also amounts to a net difference of 15 answers out of 600. The paper does not report repeated-run variability or uncertainty estimates for these comparisons. I would like to know whether the smaller trends persist across runs and balanced resamples of the existing questions, and how sensitive they are to model-output randomness and evaluation sample size.

Questions for authors
  1. How much of the FIND present/absent gap comes from the scoring requirements? Each group has 75 questions. A positive answer requires both answering Yes and providing a correct target point, whereas a negative answer requires only answering No. This asymmetry may contribute to the higher success rate on absent-target cases in Figure 3. Could you specify the point-scoring rule and separate presence-decision errors, localization errors, and episodes that terminate without an answer? This would help assess how much of the gap is due to insufficient search.

  2. Why does Astra’s performance decrease with Search? Under the Search ablation, Astra’s average success rate drops from 83.0% to 80.5%, even though manipulation occurs in 584 episodes in both settings (§4.1, p. 7). Could the authors explain this decrease? In particular, could the observed 2.5-percentage-point drop reflect sampling variability due to the limited evaluation size, rather than a systematic negative effect of the Search prompt?

  3. Why does Opus 4.8 perform worse with Search-persistent than with Search? Table 2 shows that Opus 4.8’s average success rate improves from 41.5% to 44.0% with Search, but drops back to 41.5% with Search-persistent. Meanwhile, the number of episodes involving manipulation increases from 293 to 305. Could the authors explain why performance declines despite manipulation being used in more episodes?

4 comments
AuthorsOpenReview ↗

Evaluation clarifications, FIND errors, and repeated-run results (1/2)

Thank you for your careful review and specific questions about scoring rules and result stability.

Correction in the caption: We thank the reviewer for pointing this out. The system prompt in Figure 8 correctly describes budget exhaustion: after 10 manipulation attempts, models may still invoke perception tools or answer, and exhausting the manipulation budget does not itself count as failure. Five consecutive perception-only calls trigger automatic termination and are scored as incorrect. We will correct the caption in our final version.

Response to W1/Q1:

For FIND, a target point is judged correct when the returned point lies within the ground truth target mask from the simulator in the final top-down frame, and yes, a positive answer requires both “Yes” and a correct target point to be successful. We will clarify this in our final version.

We provide a detailed breakdown of the failure cases for FIND. We categorize failures into three types: (1) Incorrect choice (VLM returned yes when ground truth is no and vice versa); (2) localization errors (VLM returned yes, ground truth is yes but 2D point does not lie on the privileged simulator mask for that termination scene step); (3) Episode termination failures (VLM used 5 perception tool calls in a row which led to automatic termination and task failure) as shown below.

ModelIncorrect Choice: Target PresentIncorrect Choice: Target AbsentLocalization Error: Target PresentLocalization Error: Target AbsentEpisode Termination: Target PresentEpisode Termination: Target Absent
Kimi K351 (68.00%)8 (10.67%)10 (13.33%)0 (0.00%)0 (0.00%)1 (1.33%)
Qwen 3.8 Flash Next42 (56.00%)25 (33.33%)19 (25.33%)0 (0.00%)0 (0.00%)1 (1.33%)
Gemini 3.1 Pro28 (37.33%)9 (12.00%)10 (13.33%)0 (0.00%)0 (0.00%)3 (4.00%)
Gemini 3.8 Flash33 (44.00%)13 (17.33%)5 (6.67%)0 (0.00%)0 (0.00%)0 (0.00%)
GPT 6 Astra9 (12.00%)14 (18.67%)6 (8.00%)0 (0.00%)0 (0.00%)0 (0.00%)

The results show that incorrect choice dominates localization errors for all models.

Response to W2:

Thank you for your suggestion. We further report repeated-run variability and uncertainty estimates on MG-VQA-Hard by measuring the mean and variance on the models’ performances across 5 seeds as shown below.

ModelCount (10)Find (10)Beneath (10)Compare (10)Average (40)Acted / 40Mean Manip./ep
Kimi K30.0±0.08.0±4.530.0±12.248.0±14.821.5±6.018.2±1.51.0±0.1
Qwen 3.8 Flash Next0.0±0.00.0±0.062.0±30.352.0±21.728.5±8.417.2±1.50.7±0.2
Gemini 3.1 Pro22.0±8.428.0±8.436.0±5.560.0±26.536.5±8.437.8±0.84.1±0.2
Gemini 3.8 Flash0.0±0.08.0±4.534.0±8.950.0±15.823.0±5.125.0±1.41.7±0.2
GPT 6 Astra20.0±7.124.0±5.536.0±5.540.0±23.530.0±8.834.6±0.53.5±0.2

Consistent with our original findings, our updated results show that all models underperform on MG-VQA-Hard compared to MG-VQA-Bench. Moreover, table-top scenes that appear uncluttered but hide hidden objects underneath the clutter expose the bias in all models to answer questions early without investigating and verifying if any evidence is hidden. We do now see that Gemini 3.1 Pro leads over Qwen 3.8 Flash Next. We will include this in our final version.

Response to Q2:

We further look into the repeated-run variability and uncertainty estimates in the Search ablation by running it on MG-VQA-Bench with three different seeds -

ModelModeCountFindBeneathCompareAverageActed / 600Mean Manip./ep
GPT 6 AstraDefault86.4 ± 1.079.3 ± 1.284.9 ± 3.476.7 ± 0.781.8 ± 1.3583.3 ± 1.53.8 ± 0.2
GPT 6 AstraSearch85.3 ± 2.077.6 ± 1.583.6 ± 0.878.4 ± 2.381.2 ± 0.7583.7 ± 1.53.7 ± 0.1

The above results show that the performance for GPT 6 Astra is statistically similar with and without the Search prompt. Additional guidance therefore produces little change in aggregate interaction frequency. We thank the reviewer for pointing this out and we will update this in our final version.

AuthorsOpenReview ↗

Evaluation clarifications, FIND errors, and repeated-run results (2/2)

Response to Q3:

We repeated the Search and Search-persistent evaluations across three runs -

ModelModeCountFindBeneathCompareAverageActed / 600Mean Manip./ep
Opus 4.8Baseline26.0 ± 0.748.2 ± 1.050.9 ± 0.844.0 ± 2.042.2 ± 0.8228.0 ± 3.61.1 ± 0.0
Opus 4.8Search27.8 ± 2.048.7 ± 0.751.3 ± 3.545.6 ± 2.343.3 ± 0.9273.0 ± 17.41.2 ± 0.1
Opus 4.8Search-persistent28.9 ± 3.250.0 ± 0.749.1 ± 3.944.9 ± 6.143.2 ± 1.6291.3 ± 13.11.3 ± 0.0

Opus 4.8 shows a narrower improvement in average success rate from baseline to search. The general finding still remains: the ‘Search’ prompt encourages the model to manipulate in more tasks. Opus achieves 43.3 ± 0.9% with Search and 43.2 ± 1.6% with Search-persistent. The repeated evaluations show a smaller increase from baseline to search and a smaller decrease in average performance with Search and Search-persistent, suggesting that the original differences reflect run-to-run variability. We will revise the discussion in the final version to emphasize that repeating the guidance provides no demonstrated additional benefit over Search in these evaluations, rather than claim that it reduces performance.

We hope the clarified protocol and additional analyses address your questions. We welcome further discussion.

AuthorsOpenReview ↗

Following up on our rebuttal

We appreciate your careful review and positive assessment. As the discussion period comes to a close, we wanted to follow up on whether our responses addressed your questions.

To recap: we have (i) clarified the Figure 8 caption: exhausting the budget does not itself end the episode, while five consecutive perception-only calls do and are scored as incorrect; (ii) specified the FIND point-scoring rule (the point must lie inside the ground-truth target mask in the final frame) and broken down FIND failures, showing that wrong presence decisions outnumber localization errors for every model; (iii) added 5-seed results on MG-VQA-Hard and will revise the ranking discussion accordingly (Sec. 4.3, Table 5); and (iv) repeated the Search and Search-persistent ablations over 3 seeds, which show that the Astra and Opus differences are within run-to-run variability. We will revise those claims in the final version (Sec. 4.3).

If our responses have resolved your concerns, we would be grateful if you would consider reflecting this in your final rating. We are glad to address anything further in the time left.

Thank you to the authors for the thoughtful clarifications and extensive additional experiments. I have carefully read the rebuttal and the other reviewers' comments, and I believe most of my concerns have been addressed.

In particular, the authors have provided additional experimental results for Table 2: Ablations conducted on MG-VQA-Bench (600 questions), offering more evidence for evaluating the findings. However, these results also seem to suggest that the ablation study does not yield sufficiently consistent conclusions, as the trends observed across Opus 4.8, Kimi K3, and GPT 6 Astra are not aligned. My interpretation is that these strong models may be relatively insensitive to additional cues, given their already high baseline performance. I believe this observation deserves a more thorough discussion in the paper.

Overall, I am still not fully convinced by the responses to Q2 and Q3, although I do not consider these remaining concerns to be major obstacles. In my view, the motivation and design of this benchmark are well aligned with the future direction of embodied AI research, which I consider a meaningful contribution.

Therefore, I have decided to maintain my rating of Weak Accept.

Reviewer_T5VSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces MG-VQA, a benchmark for answering questions about cluttered tabletop scenes through manipulation. It evaluates 8 VLMs across four tasks and examines how access to perception and manipulation tools affects their performance.

Critical strengths

The motivation is reasonable: answering questions under occlusion often requires actively gathering evidence. The environment supports simulated grasping and pushing with imperfect outcomes, and the questions are reviewed by human annotators. The evaluation also provides useful observations about premature stopping and search behavior.

Critical weaknesses
  1. The benchmark’s added scientific value is not sufficiently established. Since the questions hide answer-relevant evidence, the improvement from allowing manipulation is largely expected. The behavioral analysis is mostly descriptive: stronger models act more and re-observe more often, but the experiments do not clearly establish what drives their advantage. Final success rates combine search decisions, perception errors, manipulation failures, and answer generation. A simple clutter-clearing baseline under the same action budget, together with targeted controls on execution reliability, would help establish what the benchmark reveals beyond overall system performance.

  2. The relevance of the findings to real-world robotics remains unclear. All evaluations use a simulated tabletop environment with fixed camera views and predefined manipulation tools. The scanned assets improve visual realism, but the paper provides little evidence that the observed behaviors generalize beyond this particular simulator and tool stack. This matters especially because physical failures and recovery are central to the paper’s contribution. Without real-robot validation or stronger robustness analysis, it is difficult to judge whether the benchmark captures representative real-world limitations.

Questions for authors

How does a simple persistent clutter-clearing strategy perform under the same manipulation budget?

How sensitive are the main behavioral findings to the reliability of the perception and manipulation tools?

4 comments
AuthorsOpenReview ↗

Clutter-clearing baseline, tool controls, and real-robot results (1/2)

Thank you for your constructive feedback on the benchmark’s scientific contribution and real-world relevance. Your suggestions motivated the clutter-clearing baseline and tool-reliability controls, helping us examine what drives performance beyond access to manipulation.

Response to W1/Q1:

We agree that allowing manipulation should improve performance when answer-relevant evidence is hidden. MG-VQA’s scientific value is in evaluating how agents acquire that evidence: whether they recognize insufficient observations, select relevant interventions, and recover after failed actions. As Reviewer 43i9 noted, the setting lets researchers examine how “VLM agents respond to insufficient evidence and failed actions,” and the comparisons help distinguish models’ “willingness to use manipulation tools from their ability to effectively use the evidence obtained through manipulation.” The four tasks in MG-VQA-Bench cover skills such as hidden-object search, counting in clutter, revealing spatial relations, and identifying occluded labels. Human verification and physically consequential actions provide a common setting for examining these behaviors across eight models, beyond measuring the expected benefit of access to manipulation.

Implementing a simple persistent clearing strategy. We interpreted the requested “simple persistent clutter-clearing strategy” as a question-independent clearing policy that the VLM can invoke repeatedly before answering. Our “Declutter-only” mode replaces grasp and push with a clearing tool that uses SAM3 masks and depth to select a visible object appearing higher than others, then attempts to grasp and relocate it to an empty tabletop location. At each turn, the VLM decides whether to invoke this decluttering tool or answer the question. Our implementation of the “Declutter-only” mode is an alternate manipulation strategy that allows physical interaction with the scene to acquire hidden information. We compare this method with the original ‘Manipulation’ mode with the same maximum budget of 10 manipulation calls.

ModelModeCountFindBeneathCompareAverage
GPT 6 AstraManipulation (from Table 1)87.380.788.076.083.0
GPT 6 AstraDeclutter only60.063.364.069.364.2

The table shows that GPT 6 Astra achieves 64.2% with Declutter-only versus 83.0% with the original Manipulation mode, despite using 7.3 versus 3.9 manipulation calls per task respectively. Declutter-only reaches the ten-call budget in 334/600 episodes, compared with 38/600 for Manipulation. This simpler clearing strategy provides substantial gains over Direct and Perception modes, but does not match model-directed manipulation.

The reduced control in Declutter-only offers an explanation consistent with these results: the VLM cannot redirect object selection toward a particular unresolved occlusion or switch to pushing when grasping is ineffective. This comparison supports the utility of allowing question-conditioned action selection and alternative manipulation primitives.

If the reviewer intended a different persistent clearing policy we would welcome clarification and, where feasible during the discussion period, evaluate that variant and provide the results and analysis.

Response to W2:

We added 120 physical manipulation episodes on a Franka robot with a front-facing camera, covering two models (GPT 6 Astra and Qwen 3.8 Flash Next) and all four task categories. We provide success rate (%) below -

ModelModeCount (15)Find (15)Beneath (15)Compare (15)Average (60)
GPT 6 AstraDirect20.00.00.020.010.0
GPT 6 AstraManipulation40.053.353.366.753.3
Qwen 3.8 Flash NextDirect0.00.00.020.05.0
Qwen 3.8 Flash NextManipulation0.026.713.326.716.7

Manipulation improves GPT 6 Astra from 10.0% to 53.3% and Qwen 3.8 Flash Next from 5.0% to 16.7%, preserving the aggregate benefit of interaction and GPT 6 Astra’s advantage observed in simulation. Despite reduced clutter for hardware safety, we observed premature answers following unsuccessful interactions, including episodes ending before the ten-action allowance was exhausted. These physical trials provide supporting evidence that such limitations extend beyond the simulation experiments in MG-VQA.

AuthorsOpenReview ↗

Clutter-clearing baseline, tool controls, and real-robot results (2/2)

We also evaluated our benchmark across variations in tool and simulator configurations -

Different tool configuration. We evaluated M2T2 [1] as an alternative grasping configuration, retaining the same questions, tool access, action budget, and answer-scoring protocol. GPT 6 Astra achieves 78.8% average accuracy with M2T2, compared with 83.0% with GraspGen.

ModelGrasping ModelCountFindBeneathCompareAverage
GPT 6 AstraGraspGen (from Table 1)87.380.788.076.083.0
GPT 6 AstraM2T281.375.381.377.378.8

[1]: M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place. Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox. Proceedings of The 7th Conference on Robot Learning, PMLR 229:3619-3630, 2023.

Different simulator configuration. The Multiview experiment in the original paper’s Table 2 changes the simulator’s observation configuration by adding an oblique front-facing camera alongside the original top-down camera. Average accuracy increases from 56.7% to 62.3% for Kimi and from 41.5% to 43.7% for Opus. Improvements occur in three tasks for Kimi and two for Opus, demonstrating that the benefit of additional visual evidence is task dependent.

We study MG-VQA as the general setting where VLMs cannot answer questions directly from images, and our simulation environment allows us to understand VLM tool-calling behavior at scale while providing useful insights in how different VLMs plan, gather evidence through physical interaction, recover from failures and interpret new revealed information to answer questions in clutter.

Response to Q2:

To investigate the sensitivity of our findings to the reliability of the perception and manipulation tools, we further evaluate two models (Qwen 3.8 Flash Next and GPT 6 Astra) on MG-VQA-Bench with two additional variants. 1) Oracle perception: When the models call the perception tool, we return the ground truth scene graph and segmentation outputs from the simulator, while retaining the original manipulation calls. We note that these perception outputs only contain the visible pixels and exclude occluded objects. 2) Oracle manipulation: We retain the predicted perception outputs when the models call the perception tools but replace the robot’s execution with a ground truth removal or directional displacement of the selected object in simulation (without the robot manipulating the object) when the model calls the respective manipulation tools.

ModelModeCountFindBeneathCompareAverage
Qwen 3.8 Flash NextManipulation (from Table 1)36.742.066.071.354.0
Qwen 3.8 Flash NextOracle Perception72.076.788.762.775.0
Qwen 3.8 Flash NextOracle manipulation50.046.778.765.360.2
GPT 6 AstraManipulation (from Table 1)87.380.788.076.083.0
GPT 6 AstraOracle Perception92.092.795.379.389.8
GPT 6 AstraOracle manipulation92.082.78882.786.3

The table above shows that both errors in perception and execution contribute to end-to-end errors. Oracle perception removes failures in detecting objects and their masks while oracle manipulation removes execution failures, For Compare the VLM requires reliable badge-to-object association and three-dimensional size judgment which Qwen 3.8 Flash Next often fails at. We used state-of-the-art perception and manipulation tools (Gemini 3.6 Flash for scene graphs, SAM3 for mask generation, GraspGen for 6 DoF grasp estimation) and our released codebase supports replacing these tools with alternate perception/manipulation tools as concurrent research progresses.

We hope the new baselines and controls strengthen the evidence for our contribution. Please let us know if further clarification would help.

Thanks for the clarification. I read the rebuttal and other reviews. I think most of my concerns are addressed. I raise my score towards acceptance.

AuthorsOpenReview ↗

Response to updated review

Thank you for reading our responses and the other reviews, and for raising your score! We will include the Declutter-only baseline, comparisons with M2T2 vs GraspGen, oracle perception/manipulation controls, and the real-robot results in the final version.

Reviewer_Vw4KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MG-VQA, a benchmark for evaluating visual question answering through physical manipulation in cluttered scenes. It includes human-verified questions across four spatial reasoning tasks and an interactive simulation environment supporting perception, grasping, and pushing. Experiments on eight VLMs show that manipulation substantially improves question-answering accuracy compared with static perception. The paper also introduces MG-VQA-Hard to investigate more challenging scenarios.

Critical strengths
  1. The paper addresses a meaningful problem by evaluating whether VLMs can actively uncover hidden visual evidence through manipulation.
  2. MG-VQA-Bench provides human-verified questions across four tasks, together with an interactive environment supporting physically consequential actions.
  3. The evaluation of eight VLMs under three interaction settings provides useful insights into evidence gathering and failure recovery.
Critical weaknesses
  1. The evaluation is restricted to simulated tabletop environments with a fixed robot and predefined manipulation tools, limiting the generalizability of the findings to real-world robotic scenarios.
  2. The performance improvements may partly reflect increased interaction budgets rather than better manipulation-grounded reasoning. Stronger models generally perform more manipulation actions, but controlled comparisons under matched action budgets are missing.
  3. MG-VQA-Hard contains only 40 questions, with 10 instances per task. The small sample size limits the reliability of the conclusions.
Questions for authors
  1. How much of the performance difference between models remains when controlling for the number of manipulation actions?
  2. Have the authors evaluated whether the observed failure patterns persist across different simulator configurations or manipulation tools?
4 comments
AuthorsOpenReview ↗

Real-world evaluation, action budgets, and robustness in MG-VQA-Hard (1/2)

Thank you for your constructive feedback on generalizability, interaction budgets, and evaluation stability.

Response to W1:

We conduct real-world experiments that consist of 120 physical manipulation episodes using a Franka arm and a single front-facing camera, with 60 episodes per model across the four task categories. With the real-world front-facing camera view, we adapt our “Beneath” task to “Behind”, preserving the requirement to reveal an occluded spatial relation. We provide success rate (%) below -

ModelModeCount (15)Find (15)Beneath (15)Compare (15)Average (60)
GPT 6 AstraDirect20.00.00.020.010.0
GPT 6 AstraManipulation40.053.353.366.753.3
Qwen 3.8 Flash NextDirect0.00.00.020.05.0
Qwen 3.8 Flash NextManipulation0.026.713.326.716.7

As shown in the table compared to the Direct mode (no grasps/pushes), manipulation improves GPT 6 Astra from 10.0% to 53.3% and Qwen 3.8 Flash Next from 5.0% to 16.7%. GPT 6 Astra outperforms Qwen 3.8 Flash Next in every task under manipulation, consistent with the simulation results. We find that both models achieve lower aggregate manipulation accuracy in our real-world setup than in simulation, and in longer interaction sequences, we also observe that models answer from incomplete evidence after failed grasps or insufficient clearing showing a lot of room for future work. We share some robot rollouts here: https://anonymous.4open.science/r/mgvqa-franka-D8EB/. Our real-world experiments show that the MG-VQA task and setup is challenging even with fewer objects and clutter. We will include these findings in the final version.

Response to W2/Q1:

We would like to clarify that all the models are evaluated with the same maximum interaction budget of 10 manipulation actions per task for both MG-VQA-Bench and MG-VQA-Hard, as also specified in the system prompt (Fig. 8). Models differ in how much of this allotted budget they choose to use. This choice is part of the capability MG-VQA evaluates: an agent must decide whether the available evidence is sufficient, whether further interaction is needed, and when to stop.

To provide some comparison on “matched realized action counts” (we note that these are the number of actions used by the models as opposed to allocation), we selected rollouts from MG-VQA-Bench where three models, GPT 6 Astra, Gemini 3.1 Pro, and Qwen 3.8 Flash Next, used the same number of manipulation calls, including unsuccessful attempts. We compare their performance on this common subset of 52 questions (44 Beneath, 3 Compare, 3 Count, and 2 Find): GPT 6 Astra achieved 90.4% accuracy (47/52), compared with 75.0% (39/52) for Gemini 3.1 Pro and 78.8% (41/52) for Qwen 3.8 Flash Next with advantages of 15.4 and 11.5 percentage points, respectively. On Beneath, the largest matched category, their accuracies were 88.6%, 79.5%, and 77.3%, respectively. These results show that GPT 6 Astra's advantage persists at matched realized manipulation counts.

Response to W3:

We agree that MG-VQA-Hard is a smaller set compared to MG-VQA-Bench. We would like to point out however that this set is carefully constructed with GPT 6 Astra to utilize the failure cases from the models’ tool-calling logs in MG-VQA-Bench and also using 50 newly generated assets to guarantee that they are unseen by the models – hence its difficulty. These 40 questions are what we could achieve given the time and cost constraints at the time of submission. As this is a smaller set, we additionally provide repeated evaluation (5 seeds) of the models to provide the statistics of the models’ performance.

ModelCount (10)Find (10)Beneath (10)Compare (10)Average (40)Acted / 40Mean Manip./ep
Kimi K30.0±0.08.0±4.530.0±12.248.0±14.821.5±6.018.2±1.51.0±0.1
Qwen 3.8 Flash Next0.0±0.00.0±0.062.0±30.352.0±21.728.5±8.417.2±1.50.7±0.2
Gemini 3.1 Pro22.0±8.428.0±8.436.0±5.560.0±26.536.5±8.437.8±0.84.1±0.2
Gemini 3.8 Flash0.0±0.08.0±4.534.0±8.950.0±15.823.0±5.125.0±1.41.7±0.2
GPT 6 Astra20.0±7.124.0±5.536.0±5.540.0±23.530.0±8.834.6±0.53.5±0.2

Consistent with our original findings, our updated results show that all models underperform on MG-VQA-Hard compared to MG-VQA-Bench. Moreover, table-top scenes that appear uncluttered but hide hidden objects underneath the clutter expose the bias in all models to answer questions early without investigating and verifying if any evidence is hidden. We do now see that Gemini 3.1 Pro leads over Qwen 3.8 Flash Next. We will include these results in our final version.

AuthorsOpenReview ↗

Real-world evaluation, action budgets, and robustness in MG-VQA-Hard (2/2)

Response to Q2:

Different tool configuration. We additionally evaluated M2T2 [1] as an alternative grasping configuration, retaining the same questions, tool access, action budget, and answer-scoring protocol. GPT 6 Astra achieves 78.8% average accuracy with M2T2, compared with 83.0% with GraspGen.

ModelGrasping ModelCountFindBeneathCompareAverage
GPT 6 AstraGraspGen (from Table 1)87.380.788.076.083.0
GPT 6 AstraM2T281.375.381.377.378.8

[1]: M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place. Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox. Proceedings of The 7th Conference on Robot Learning, PMLR 229:3619-3630, 2023.

Different simulator configuration. The Multiview experiment in the original paper’s Table 2 changes the simulator’s observation configuration by adding an oblique front-facing camera alongside the original top-down camera. Average accuracy increases from 56.7% to 62.3% for Kimi and from 41.5% to 43.7% for Opus. Improvements occur in three tasks for Kimi and two for Opus, demonstrating that the benefit of additional visual evidence is task-dependent.

Consistency of failure and recovery patterns. GPT 6 Astra’s dominant recovery strategy persists: following the first grasp call that fails to place an object, the next manipulation is a push in 82.7% of failure-containing episodes with GraspGen and 79.9% with M2T2. Identification, volume-comparison, and final-point errors also occur after successful grasps, showing that the manipulation tool alternate still retains challenges in spatial reasoning post manipulation. The Multiview results show that additional visual evidence through a second camera input improves some tasks but leaves substantial errors across all four categories.

We hope these additional experiments address your concerns, and we welcome follow-up questions during the discussion.

AuthorsOpenReview ↗

Following up on our rebuttal

Thank you again for your review and the positive assessment. With the discussion period closing, we wanted to check whether our responses resolved your concerns.

To recap: (i) We added real-world experiments, with 120 manipulation episodes on a Franka arm. These keep both the gain from manipulation (Astra 10.0% → 53.3%, Qwen 5.0% → 16.7%) and the ordering between models seen in simulation. (ii) We clarified that all models share the same 10-action budget, so how much of it they use is part of what is evaluated in MG-VQA. On questions where GPT 6 Astra, Gemini 3.1 Pro and Qwen 3.8 Flash Next used the same number of manipulation actions, Astra still leads (90.4% vs. 75.0% and 78.8%, respectively). (iii) We added 5-seed means ± SD on MG-VQA-Hard, and will revise the MG-VQA-Hard discussion (Sec. 4.3, Table 5) to reflect the updated rankings. (iv) We showed robustness of our findings to a different grasping model (M2T2: 78.8% vs. GraspGen: 83.0%), with the same recovery pattern (push after a failed grasp in M2T2: 79.9% vs. GraspGen: 82.7% of episodes where a grasp failed) when evaluated with Astra as the tool-calling VLM.

If our responses have resolved your concerns, we would be grateful if you would consider reflecting this in your final rating. We are glad to address anything further in the time left.

Thanks for the authors' detailed response. I have decided to maintain my rating of Weak Accept.

Comments on the paper

1 comment

Dear reviewers, the authors have posted their responses to your reviews. Pls have a look and discuss.

Thanks, AC for #21924

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.