- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EMBRACE, a benchmark for testing whether vision-language models adapt to visual conventions that differ across jurisdictions. The benchmark covers 10 object families in healthcare, manufacturing, and retail, with 286 family-jurisdiction mappings. The authors evaluate four models under conditions where the local convention is present or absent. They also use a jurisdiction-blind baseline to measure how much model behavior depends on the stated jurisdiction. The results show that models often follow a preferred convention instead of the local convention. They also often select a foreign convention when no locally correct option is available.
Critical strengths
The main strength is the problem itself. The paper studies an important and underexplored failure mode in embodied models: the same visual cue can have different meanings in different jurisdictions. This issue can matter in safety-relevant settings such as healthcare and manufacturing. The benchmark gives a clear way to test whether models use the stated jurisdiction instead of relying on a fixed visual prior.
A second strength is the controlled evaluation design. The paper separates cases where the local convention is present from cases where it is absent. It also compares model behavior with a jurisdiction-blind baseline. This design makes the main failure mode easy to interpret.
Critical weaknesses
The main weakness is the limited visual coverage and uncertain validity for real deployments. The benchmark uses 89 images, and all benchmark images pass through an editing pipeline. The authors also remove or obscure object text so that color is the main jurisdiction-specific cue. This design gives good experimental control, but it may not represent the visual diversity and contextual information available in real workplaces.
This issue limits how strongly the results can support claims about real embodied systems. The benchmark clearly demonstrates the failure mode in a controlled setting, but it is less clear how often the same failure occurs with natural, unedited images in realistic deployment conditions. The authors also identify this limitation in the paper.
Questions for authors
None
2 comments
Thank you for the detailed response and additional experiments. I appreciate the clarification of the image construction process and the new results with explicit color rules, which help address my original concerns. While real-world generalizability remains a limitation, these revisions are helpful. My overall assessment remains positive, and I will maintain my original rating.
Rebuttal by Authors
Thank you for your positive review, and especially for recognising the importance of our problem (cross-jurisdiction convention failures in safety-relevant settings) and the clarity of our controlled evaluation design.
We agree with you that editing colours and reducing text could limit the realism overall, and we want to add a bit more context (both here and in the revised paper) on how we created our images dataset.
All 89 benchmark images originate from 18 original photographs taken by the authors (nine single-object and nine seven-object base photos). For example, the blood-tube image was taken by one of the authors during a routine blood check and the pictures of fire extinguishers were taken in our workspace. Whenever it was possible, we tried to limit the visibility of the confounding text by taking a picture from a different angle rather than digitally erasing it. For example, we chose a hospital oxygen cylinder whose printed label had worn off with time, and photographed a fire extinguisher from an angle that did not show the full description on the body. We’ve done it also to reflect a realistic and a practical deployment - as a robot in the field can encounter equipment where text is worn, facing away, or occluded by a rack, leaving colour as the primary visible cue. Using these base photographs, we changed only the regulated colour regions while keeping the rest of the setting stable, and every edited image needed the highest rating in realism both from a human annotator and an autorater. Following up on your feedback, we will revise imprecise wording in Section 3.4 ("adding or removing objects") and Appendix A.2 ("synthetic benchmark images") so this pipeline is described clearly.
To check if the difficulties in recognising or locating the edited objects could explain our results, we ran an additional experiment on the identical seven-object edited images in which the prompt explicitly states the local colour convention (we added the sentence along the lines of 'The color of X in country Y is Z' with the correct information for each prompt). Please note that unfortunately we came across technical issues with the Fable API, which we won’t manage to resolve in time for the rebuttal, but will have it ready for the camera ready version. We also added Gemini 4 Argon to our comparison and plan to expand the number of evaluated models.
n = 268):Local ConventionaccuracyNo Selection(Abstention)On four of five models, explicitly providing the local convention brings Condition C accuracy close to 100%, suggesting that difficulties recognising or locating the edited objects alone cannot explain the observed failures.
When it comes to the size of the dataset, we selected the ten object families from a broader pool of candidate conventions across three industries and intentionally kept the core image set compact (89 edited images) so that the benchmark remains financially and computationally accessible across 1,126 jurisdictional prompts per grounding format, or 2,252 evaluations per model across pointing and bounding boxes. For the camera-ready version, we also plan to release all images, prompts, convention keys, and evaluation scripts, along with examples of unedited real-world photographs and visible workplace text, so practitioners can use our framework on their own environments and evaluations in safety settings.
We agree that our controlled benchmark does not establish how frequently these failures would occur with unedited images in real workplaces, but we think it is an important first step in isolating and measuring this failure mode, which has not been covered by prior work.
We thank you again for your careful reading and constructive feedback which helped us improve our paper, and we would be very grateful if you would consider updating your score in light of these clarifications and additional results.