- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper construct a VQA benchmark for tabletop robot manipulation tasks. The benchmark design is focus on open-ended questions such as pointing and bounding box detection during robot task executions. This paper evaluates state of the art VLMs including the close source model GPT6 astra, and open source model such as Qwen family. The paper finds that the specialized task training would potentially degrade model performance in general tasks.
Critical strengths
- The propose VQA benchmark uses open-ended format. This format is practical and aligns closely to QAs task in robot deployment.
- The benchmark uses data collected from real robot execution.
- The related work section has a good coverage of the prior work.
Critical weaknesses
- The paper does not clearly present the data distribution, such as what are the robot task categories used to generate the VQA. From the current description, the data mostly comes from a single setup, the washing-machine scene. If so, this paper does not provide sufficient coverage of tabletop manipulation task. Without clearly description of the dataset, it is hard to evaluate the soundness of the conclusion.
- The paper's taxonomy, Table 2, has been mostly covered by many existing benchmarks such as ManipBench, ManipVQA, Robo2VLM, etc. This paper lacks unique question design and perspective to reveal the model's capability and limitations.
- The findings are generic and lack depth. Most of the finding and observation are already covered by existing papers. Since the author focus on deployment and the open-ended format, it would be more interesting to find the relationship between spatial grounding answer variance and robot task success rate.
Questions for authors
Please address questions in the weaknesses section. The most important question for this paper is: what capability does this benchmark measure that cannot already be covered by existing embodied/robot VQA benchmarks?
2 comments
Follow-up to Reviewer wKgs
Dear Reviewer wKgs,
I am writing to gently follow up on our earlier message. If you have had a chance to read our response, we would be very grateful to know whether it addresses your concerns. If anything is still unclear, we would be glad to clarify it promptly.
Thank you again for your time and for your careful review.
Best regards, The Authors
Follow-up on Our Response
Dear Reviewer wKgs,
Thank you again for your thoughtful review. In our response, we explained what our dataset covers, compared our benchmark with existing benchmarks, and shared results from a real-robot experiment.
If you have had time to read our response, could you please let us know whether it addresses your concerns? We are happy to answer any further questions.
Thank you for your time.