acceptodds
Under review as a conference paper at ICLR 2027

RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

Abstract

Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability for agents is to be able to understand what the worker is doing, seeing through the human’s camera or smart glasses, and assist them in documentation. Despite impressive advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work scenarios in Blender across eleven distinct environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to the real images.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, benchmark, multimodal evaluation, synthetic data, sim-to-real, fine-grained visual perception, image quality assessment, egocentric vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NZzeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

RealityBench provides 11 Blender environments (1,378 frames) for rooftop solar installation (rail_attachment, wire_adequacy, tape_gap, slope_gauge, equipment_label, horizon_peak, breaker_label) and vehicle repair (car_view, damage_view, car_plate, ro_board). It evaluates whether models can extract relevant information from them to document the task. The benchmark evaluates 26 VLMs using programmatic scorers checked against human judgments. It also compares synthetic and real-image performance on three documentation questions (tape_gap, car_view, damage_view) and analyzes where failures arise from scene (e.g., misreading the requirement or scene).

Critical strengths
  • Camera pose, visibility, blur, and geometry are exposed, enabling investigations of model behavior. Covers 26 VLMs and examines failure patterns.
  • Photograph acceptability is checked against human judgments, with held-out validation and ambiguous cases excluded. Authors test if defects change when acceptability verdict is asked first/last/omitted.
Critical weaknesses
  • Only two work settings are covered (rooftop solar installation and vehicle repair) but physical work is more diverse. Tasks are also static (e.g., identifying the car view, damage severity, plate number, etc., without observing repair or correction).
  • Real-world validation covers only 3 of the 11 tasks. Holding a tape measure against everyday objects tests measurement reading, but is still far from documenting actual installation or repair under varying weather/viewpoints/camera conditions.
Questions for authors
  • Could some low scores, particularly for damage severity, come from ambiguous labels? With area-based categories, judging damage from a photo would be subjective, I can imagine?
Reviewer_s5UDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces RealityBench, a benchmark for evaluating whether VLMs can assess the adequacy of photographic documentation of physical work and extract task-relevant information. The benchmark comprises 11 Blender environments spanning multiple scenarios. The authors evaluate a range of models on the generated images and compare performance on synthetic and real images across three documentation questions. The results show persistent challenges in reading measurements, recognizing small parts, and making spatial judgments of recent VLMs.

Critical strengths
  • Distinguishing photograph acceptability from information extraction is an important task, capturing aspect of real-world documentation that conventional VQA benchmarks may overlook.
  • Procedural environments enable generating diverse scene properties, camera poses, and image quality, while providing inspectable geometric and object-state annotations without relying on a LLM-based judge.
  • The analysis provides insights beyond aggregate rankings.The paper also includes different models, uncertainty estimates, human assessments, which is relative comprehensive.
Critical weaknesses
  • The scorer relies on manually specified interpretations and thresholds and is validated on only 95 held-out frames. Agreement is relatively low in some environments, even among human judges, and four of the six annotators are authors., which may lead to bias. Disagreement with the scorer therefore does not necessarily imply an incorrect judgment.
  • The transfer study covers only three documentation questions and excludes the acceptability task. While the ranking correlations are informative, they neither establish transfer across the full benchmark nor demonstrate comparable absolute performance on synthetic and real images.
  • The evaluation is narrower than the paper’s broader motivation of developing assistants for physical work. All experiments use individual images, without temporal context, active observation, or closed-loop guidance (i.e., videos might be a better surface format). Moreover, documentation performance is primarily evaluated on images deemed acceptable by the scorer, limiting the generalizability of the claim that text reading is largely solved.
Questions for authors
  • How sensitive are model rankings to scorer thresholds and gray-zone definitions? Please report sensitivity analyses and, if feasible, validation with independent domain experts.
  • Could the real-image evaluation include acceptability judgments and additional environments, particularly small-part recognition and contact relationships?
  • Human performance also appears relatively low. To what extent does this reflect ambiguity in the task requirements or evaluation criteria, rather than the inherent difficulty of the task?
Reviewer_b7ZuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

RealityBench evaluates whether vision-language models can judge photographs as adequate documentation of physical work and extract task-relevant information from them. The benchmark uses eleven Blender environments covering rooftop solar installation and vehicle repair, with programmatic scorers calibrated against human ratings. It evaluates 26 models and compares synthetic and real-image performance on three documentation questions. The results identify measurement reading and small-part or spatial judgments as persistent difficulties, while also examining disagreement in photograph acceptability.

Critical strengths

(1) I think the problem this paper investigate is a good problem and this is very important for agent foundation model to effectively work in physical world. Task-conditioned photograph acceptability is also a useful problem that is distinct from generic image quality or object recognition. (2) The procedural environments provide controlled scene variation and access to scene state, making it possible to investigate specific failure factors rather than only report aggregate benchmark scores. (3) The evaluation includes meaningful validation beyond synthetic model scores. The paper reports human agreement, paired confidence intervals, real-photograph comparisons, and verdict-order ablations. It also explicitly acknowledges limitations of the human reference and the restricted scope of the real-world comparison.

Critical weaknesses

(1) The real-image ranking-transfer evidence is affected by model-dependent exclusions. Table 14 excludes tape-measure responses truncated by the output cap, including 54 responses for GLM-5.3 Flash. Models therefore need not be scored on the same real-image instances, and truncation may be related to question difficulty. The transfer result is also limited to three documentation questions and provides no direct validation of real-world photograph acceptability, so it cannot establish transfer of the benchmark as a whole. (2) The acceptability reference remains uncertain for several environments. Four of the six raters are authors, the human comparison uses non-experts, and the frozen scorer is validated against majority decisions on 95 held-out frames in total. Table 13 shows low chance-corrected agreement for some tasks, particularly damage_view and horizon_peak. Agreement with a tuned scorer should therefore not be interpreted as equivalence to professional inspection judgment, and the sensitivity of model rankings to plausible alternative interpretations remains important.

Questions for authors

(1) Do the reported synthetic-to-real rank correlations persist when all truncated responses are counted as failures, or when those requests are rerun with a sufficient output cap? Please provide a comparison on a common instance set across models and uncertainty estimates for the resulting correlations. (2) How sensitive are acceptability rankings to scorer thresholds, exclusion of ambiguous frames, and the use of author versus non-author judgments? Please report results on the common held-out human-rated subset against both the frozen scorer and the human reference. (3) The main text states that the horizon-level requirement was removed from acceptability scoring, but Table 1 still lists a roll threshold. Which specification produced the reported results? Please provide the final prompt-to-scorer mapping, including requirements that are deliberately not checked. (4) The relevant works regarding evaluating physical vision in simulated enviroments could be included into discussion: IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering; RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments. (5) Can the real-image damage-severity labels be independently checked on the existing image set, given that they currently come from one author and the synthetic definition is based on surface area? This would help determine whether the observed transfer reflects the intended judgment rather than differences in label interpretation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.