- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
RealityBench provides 11 Blender environments (1,378 frames) for rooftop solar installation (rail_attachment, wire_adequacy, tape_gap, slope_gauge, equipment_label, horizon_peak, breaker_label) and vehicle repair (car_view, damage_view, car_plate, ro_board). It evaluates whether models can extract relevant information from them to document the task. The benchmark evaluates 26 VLMs using programmatic scorers checked against human judgments. It also compares synthetic and real-image performance on three documentation questions (tape_gap, car_view, damage_view) and analyzes where failures arise from scene (e.g., misreading the requirement or scene).
Critical strengths
- Camera pose, visibility, blur, and geometry are exposed, enabling investigations of model behavior. Covers 26 VLMs and examines failure patterns.
- Photograph acceptability is checked against human judgments, with held-out validation and ambiguous cases excluded. Authors test if defects change when acceptability verdict is asked first/last/omitted.
Critical weaknesses
- Only two work settings are covered (rooftop solar installation and vehicle repair) but physical work is more diverse. Tasks are also static (e.g., identifying the car view, damage severity, plate number, etc., without observing repair or correction).
- Real-world validation covers only 3 of the 11 tasks. Holding a tape measure against everyday objects tests measurement reading, but is still far from documenting actual installation or repair under varying weather/viewpoints/camera conditions.
Questions for authors
- Could some low scores, particularly for damage severity, come from ambiguous labels? With area-based categories, judging damage from a photo would be subjective, I can imagine?