acceptodds
Under review as a conference paper at ICLR 2027

What a Robot Harness Asks Its VLM: A Deployment-Grounded Benchmark

Abstract

The release of GPT-6 Astra has intensified interest in embodied systems built around vision–language models (VLMs) and robot-specific harnesses. In such systems, however, VLMs are still commonly evaluated as standalone models rather than through the interfaces at which they operate. Even existing benchmarks grounded in real-world embodied settings typically measure either end-to-end system performance or isolated capabilities such as spatial reasoning, abstracting away the interface through which a robot software stack queries its VLM. Their results therefore provide limited guidance for model selection and diagnosis in deployed embodied systems. We address this gap by introducing and releasing a deployment-grounded benchmark derived from requests logged during the routine operation of a manipulation robot already deployed in a real-world environment. Rather than constructing tasks for evaluation, we derive both the task taxonomy and the evaluation instances from actual calls issued by the robot stack. The benchmark covers eight request categories and preserves the task-relevant processed views and output contracts used in deployment. It contains category-balanced, semantically aligned Chinese–English question pairs. We evaluate representative VLMs with output-contract-specific metrics that distinguish format violations from semantic errors. This setup enables a deployment-grounded comparison of VLMs at the robot–VLM interface through which they are actually used.

Primary area:
datasets and benchmarks
Keywords:
Datasets, Benchmark, VLM, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wKgsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper construct a VQA benchmark for tabletop robot manipulation tasks. The benchmark design is focus on open-ended questions such as pointing and bounding box detection during robot task executions. This paper evaluates state of the art VLMs including the close source model GPT6 astra, and open source model such as Qwen family. The paper finds that the specialized task training would potentially degrade model performance in general tasks.

Critical strengths
  1. The propose VQA benchmark uses open-ended format. This format is practical and aligns closely to QAs task in robot deployment.
  2. The benchmark uses data collected from real robot execution.
  3. The related work section has a good coverage of the prior work.
Critical weaknesses
  1. The paper does not clearly present the data distribution, such as what are the robot task categories used to generate the VQA. From the current description, the data mostly comes from a single setup, the washing-machine scene. If so, this paper does not provide sufficient coverage of tabletop manipulation task. Without clearly description of the dataset, it is hard to evaluate the soundness of the conclusion.
  2. The paper's taxonomy, Table 2, has been mostly covered by many existing benchmarks such as ManipBench, ManipVQA, Robo2VLM, etc. This paper lacks unique question design and perspective to reveal the model's capability and limitations.
  3. The findings are generic and lack depth. Most of the finding and observation are already covered by existing papers. Since the author focus on deployment and the open-ended format, it would be more interesting to find the relationship between spatial grounding answer variance and robot task success rate.
Questions for authors

Please address questions in the weaknesses section. The most important question for this paper is: what capability does this benchmark measure that cannot already be covered by existing embodied/robot VQA benchmarks?

2 comments
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer wKgs,

Thank you again for your thoughtful review. In our response, we explained what our dataset covers, compared our benchmark with existing benchmarks, and shared results from a real-robot experiment.

If you have had time to read our response, could you please let us know whether it addresses your concerns? We are happy to answer any further questions.

Thank you for your time.

AuthorsOpenReview ↗

Follow-up to Reviewer wKgs

Dear Reviewer wKgs,

I am writing to gently follow up on our earlier message. If you have had a chance to read our response, we would be very grateful to know whether it addresses your concerns. If anything is still unclear, we would be glad to clarify it promptly.

Thank you again for your time and for your careful review.

Best regards, The Authors

Reviewer_hMc3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds a benchmark from calls that a binocular-camera manipulation harness issued to its VLM during real operation in tabletop-manipulation and washing-machine scenes. It samples 1,200 requests, one from each of 1,200 distinct successful episodes, with 150 requests in each of eight request families. Each item keeps the visual payload sent to the VLM and the output contract, and comes with aligned Chinese and English prompts. References are proposed independently by two models; requests on which the proposals disagree, agree only in part, or have no agreement score are adjudicated by a human. The evaluation reports scorer acceptance separately from reference-relative correctness. The authors evaluate 15 VLMs on the same 2,400 prompts. The main finding is that none of the four embodied specialists has a higher overall request-score point estimate than its matched base. In particular, Embodied-R1.5-8B leads Qwen3-VL-8B on three published embodied dimensions but trails it by 4.96 points on these requests.

Critical strengths
  1. The evaluation is placed at the actual interface between the harness and the VLM. Items are taken from real robot-issued calls and keep processed crops, conditional requests and output contracts, and the paper reports contract acceptance separately from the request score. This lets the benchmark reflect what a modular robot system actually requires of its VLM, and offers a valuable perspective on model selection. The paper also gives a practically meaningful caution: with the interface held fixed, an embodied specialist that leads its base on published benchmarks can score lower than that base on these robot-issued requests.

  2. The statistical analysis and reporting are careful. The authors use a paired bootstrap over episodes that preserves the sampling strata, label the family-level comparisons as exploratory, and provide a majority-class constant baseline and sensitivity analyses. The references were re-annotated rather than taken from the deployed model's replies. The internal numbers I checked, as well as the published scores reproduced from other papers, are consistent. I therefore consider the observed score gaps credible under the current data and evaluation protocol.

Critical weaknesses
  1. The paper presents the benchmark as a tool for model selection and diagnosis in deployed embodied systems, but does not say whether it is meant for the authors' own system or is expected to generalize to other systems. These two uses require different evidence. For model selection in a specific system, the overall score should be weighted by how often each request family is actually called in operation, rather than averaging the eight families equally. Physical-robot experiments would also be needed to test whether request scores predict task success, or are at least correlated with it. The physical executions in the paper run only the references, which establishes that the references are executable but says nothing about how different models perform on real tasks. The authors list this as future work in the discussion, but without it the model-selection use lacks direct support. In addition, all data come from a single harness and only from successful episodes. The paper names the two scene types, tabletop manipulation and washing machine, but does not report the number of tasks or object combinations, or the share of each scene type. This makes it hard to treat the benchmark as a general one; it is closer to a case study of a single system. As for the main finding, that no embodied specialist has a higher overall request score than its base, I see two problems. First, only four matched pairs are compared, which is a small number. Second, although the authors report acceptance and the score on accepted responses separately, model capability still cannot be separated from format compliance, so the conclusion is not yet reliable (see Weakness 2).

  2. The current evaluation does not adequately separate a model's task capability from its format compliance, so differences in the overall request score cannot be attributed directly to differences in capability. All models receive the same prompts derived from production (verbatim or lightly clarified), and the deployed VLM for which these prompts were written is not identified beyond not being a Qwen3-VL model. Rejected responses score zero, yet the paper does not analyze how many of them actually contain a correct decision. The authors acknowledge that neither acceptance nor the score on accepted responses separates formatting failures from errors of perception or reasoning. In Appendix D.4, when comparing Chinese and English prompts, the paper already restricts the analysis to pairs accepted in both languages, and the differences shrink or even reverse: Qwen3.5-9B falls from 5.98 to 2.47 points, Qwen3-VL-8B from 2.91 to 2.14, and Qwen3.5-2B goes from +1.90 to −0.48. This shows that acceptance alone can substantially change apparent differences on this benchmark, but the authors do not apply the same analysis to the specialist–base pairs. The conclusion that embodied specialists underperform their bases therefore needs to be verified on the requests accepted for both models in each pair before it can stand.

Questions for authors
  1. Please clarify whether the benchmark is intended for model selection in your own system or is meant to generalize to other systems. Please also report how often each request family is called in actual operation, together with the overall score and model ranking when the families are weighted by these frequencies.

  2. Please report the number of tasks and object combinations covered by the 1,200 episodes and the share of tabletop-manipulation and washing-machine scenes, and state when the full release will be available and under what license.

  3. Please state the model family of the deployed VLM. Following the approach of Appendix D.4, please also report the specialist-minus-base difference for each matched pair on the requests accepted for both models, and manually inspect a sample of invalid responses to estimate how many of them contain a correct decision. If the results show that the gaps come mainly from format compliance, I will adjust my rating accordingly.

Reviewer_bMd1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a deployment grounded benchmark for evaluating VLMs at the robot–model interface, where the robot harness determines requests, visual inputs, and response formats. It comprises requests across eight task families, with aligned English–Chinese prompts and scoring that captures both response-contract compliance and correctness. Evaluation of 15 VLMs reveals substantial variation across task families and shows that all four embodied specialists underperform their corresponding base models. These findings suggest that gains on existing embodied benchmarks may not transfer to the requests issued by deployed robot systems.

Critical strengths
  1. The benchmark is built from real deployed robot-harness calls rather than from common evaluation tasks.
  2. The reference answers were created by two models, with humans resolving the disagreements, and double-checked on a subset, which makes the scoring credible.
  3. The design of the experiment produces unexpected, practical results: a fine-tune that outperforms its base on public benchmarks scores lower than that base on real robot requests.
Critical weaknesses
  1. The benchmark draws exclusively from successful episodes of a single company’s robot harness, limiting generalizability to other systems and failure-recovery settings. The undisclosed collection model also makes potential model–harness co-adaptation difficult to assess.
  2. Although acceptance and conditional correctness are reported separately, the evaluation does not isolate contract failures from perception or reasoning errors. Retaining Chinese target literals in English prompts also prevents the language comparison from isolating prompt-language effects.
Questions for authors
  1. Since S|V is computed on each model's own accepted answers, would the overall ranking hold on the common subset of requests accepted by all models?
  2. Do the specialists lose because of weaker perception or because of poor contract adherence? Would the specialist–base gap shrink if the output schema were stated explicitly in the prompt or after a minimal format calibration?
  3. Is it possible that the request distribution and output contracts are implicitly co-adapted to the unnamed model that served the harness in production?

Comments on the paper

1 comment
Area_Chair_39uWOpenReview ↗

[To Reviewers] Active Participation in the Author–Reviewer Discussion

Dear Reviewers,

As the author–reviewer discussion period is coming to end soon, this is a gentle reminder to stay engaged. Please take some time to review and acknowledge the authors’ rebuttals, follow up on any remaining questions, and update your reviews as appropriate.

Thanks,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.