- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldAuditBench, a benchmark for evaluating multimodal agents on anomaly detection in interactive 3D environments. It contains 213 tasks across 13 environments implemented in Unreal Engine 5 and Three.js. The tasks cover five anomaly families, including physical, spatial, temporal, and semantic inconsistencies.
The authors evaluate two approaches: an interactive VLM agent that reasons and acts during exploration, and a two-stage VLA–VLM system that explores first and analyzes the recorded trajectory afterward. Five model backbones are evaluated, together with a human baseline. The results show that current agents perform substantially worse than humans, and interactive VLM agents generally outperform the two-stage approach. The paper also analyzes anomaly exposure, task guidance, exploration budgets, and multiple anomalies.
Critical strengths
-
The benchmark addresses a useful problem that is not well covered by existing embodied AI benchmarks. Some anomalies, such as missing collision or disappearing objects, cannot be reliably identified from a single image. They require interaction or comparison across observations. The inclusion of different anomaly types and interactive environments makes this benchmark potentially useful for evaluating more realistic agent behavior.
-
The dataset construction and analysis require considerable effort. The authors manually create the anomalies and use independent human validation to check their quality. I also find the analysis of anomaly exposure versus identification useful. It helps distinguish whether an agent fails because it never encounters the anomaly or because it cannot identify the anomaly after observing it. This is more informative than reporting success rates alone. The project page is also well designed, with an interactive demo that makes the benchmark tasks more accessible and easier to understand.
Critical weaknesses
-
The evaluation setting makes the auditing task considerably easier than general anomaly discovery. Each main evaluation task contains exactly one target anomaly, and agents receive an anomaly-type hint and a matching in-context example. The ablation in Table 3 shows that removing these hints reduces the VLM success rate from 33.3% to 15.9% on Unreal Engine tasks. This suggests that the results depend substantially on knowing what kind of anomaly to search for. There are also no normal environments in the main evaluation, so false positive behavior is not measured. In addition, the evaluation protocol does not consistently require evidence from the agent's trajectory to verify the reported anomaly. Unless explicitly required by the rubric, a semantically matching report may receive full credit without sufficient visual evidence of the claimed anomaly. This makes it harder to determine whether the agent actually observed and verified the anomaly.
-
The comparison between the two auditing paradigms does not sufficiently isolate the effect of reasoning-guided exploration. In the VLM-only setting, the evaluated VLM backbone directly selects exploration actions, while the VLA–VLM setting uses the same Open-P2P 1.2B model for exploration across all backbones. The two approaches also have different exploration budgets and available reasoning capabilities. Thus, the performance gap cannot be attributed only to coupling reasoning with action. The results in Appendix D further show that the two-stage approach performs better for several backbones on Three.js environments, which suggests a strong dependence on the environment and exploration policy. I think this limits one of the paper's main conclusions.
Post-rebuttal Update
The authors have provided substantial additional experiments addressing my original concerns. These results strengthen the evaluation and clarify several limitations of the original benchmark. Some concerns remain regarding the scope of open-ended auditing and the interpretation of the comparison between auditing paradigms. However, the additional evidence addresses my main methodological concerns.
I have therefore updated my rating from 2 to 3. I appreciate the authors' detailed responses and efforts during the discussion.
Questions for authors
-
How much would the reported success rates change if agents were evaluated without anomaly-type hints and with normal scenes included? In particular, can the authors report false positive rates and evidence-grounded success rates? The existing guidance ablation is helpful, but it does not fully address this issue.
-
Can the authors provide a more controlled comparison between online reasoning and offline trajectory analysis, using the same exploration backbone or a matched exploration budget? This would help determine whether the advantage comes from online reasoning or from the different exploration policies.
-
How often do agents receive a successful score when their trajectories do not contain sufficient evidence to verify the reported anomaly? Would requiring such evidence change the ranking between methods? I understand that some rubrics already impose evidence requirements, but it is unclear how consistently this is enforced across the benchmark.
10 comments
Response to Reviewer NYUB (2 / N)
W1.3. Evidence-grounded evaluation. (Also part of Q1 and Q3)
We conduct an additional evidence-grounded evaluation using an updated judge with stricter visual evidence requirements. Specifically, we modify the evaluation protocol in Appendix C.4 in three ways:
- Mandatory visual evidence: A semantically correct report alone is no longer sufficient for a successful score. The reported anomaly must also be supported by the submitted visual evidence.
- Evidence sufficiency: The evidence must demonstrate the reported inconsistency. For anomalies involving changes or physical interactions, the submitted evidence must contain the necessary observations to verify the anomaly, such as before- and after-event frames.
- Stricter scoring: Reports with missing, irrelevant, or insufficient visual evidence receive a score of 0, even if their textual descriptions match the target anomaly.
We also report human eval results to show that our updated judge aligns with human judgment.
| Backbone | Method | Original score | Evidence-grounded score (by model) | Evidence-grounded score (by human) |
|---|---|---|---|---|
| GPT-6 Astra | VLM | 42.3% | 42.3% | 42.3% |
| GPT-6 Astra | VLM–VLA | 16.4% | 14.6% | 16.4% |
| Gemini 3.8 Flash | VLM | 32.4% | 31.0% | 31.9% |
| Gemini 3.8 Flash | VLM–VLA | 17.4% | 15.5% | 16.4% |
| Claude Opus 5 | VLM | 28.2% | 27.7% | 27.7% |
| Claude Opus 5 | VLM–VLA | 15.0% | 13.6% | 14.6% |
Human assessment reduces success rates by 0.0–1.0 percentage points, while model-based assessment reduces them by 0.0–1.8 percentage points. These results suggest that receiving a successful score without sufficient supporting evidence in the observed trajectory is uncommon. In our human review, the few errors mainly concern evidence selection: omitting a necessary before- or after-event frame, or including irrelevant frames, which we count as errors. When a necessary frame is omitted from the selected evidence, it is still actually available in the full trajectory.
Response to Reviewer NYUB (3 / N)
W2. Fairness of the comparison between auditing methods. (Also Q2)
We examine comparison fairness along three dimensions: exploration distance, model backbones, and inference costs.
The main evaluation uses 40 actions for the VLM and a 60-second exploration setting for the VLA, aiming for roughly comparable exploration distances. We will clarify this comparison in the revised PDF.
To address the remaining two dimensions, we additionally conduct matched-backbone experiments and an inference-cost budget analysis, as detailed below.
Below is the result for the matched-backbone experiments:
| Setting | GPT SR |
|---|---|
| A. VLM agent (GPT) | 42.3% |
| B. GPT free exploration → GPT analysis | 28.6% |
| C. GPT anomaly-aware exploration → GPT analysis | 32.4% |
| D. VLA exploration → GPT analysis | 16.4% |
Each entry covers all 213 tasks. B and C share the same exploration harness, tools, and 40-action budget, differing only in the exploration objective. Both separate exploration and analysis into two roles: an explorer collects observations, and a fresh GPT analyzer reviews the resulting trajectory with the original task context.
B and C outperform D, suggesting a capability gap between the current VLA explorer and GPT-based exploration. A further outperforms B and C, consistent with the benefit of keeping analysis and exploration coupled: its auditor can gather additional evidence as new hypotheses emerge, whereas the separate analyzers in B and C can only review the completed trajectories.
We also aim to control for inference cost. Settings A, B, and C already incur similar costs, whereas D is substantially cheaper. We therefore increase D’s budget through longer exploration recordings and additional offline review rounds, aiming to bring its inference cost closer to that of A–C.
| Setting | SR | Mean inference cost/task |
|---|---|---|
| A. VLM agent (GPT) | 42.3% | $2.955 |
| B. GPT free exploration → GPT analysis | 28.6% | $2.814 |
| C. GPT anomaly-aware exploration → GPT analysis | 32.4% | $3.015 |
| D1. VLA exploration → GPT analysis (non cost-matching) | 16.4% | $0.281 |
| D2. VLA exploration → GPT analysis (cost-matching) | 17.4% | $3.681 |
A outperforms B and C at similar inference costs. For D, longer exploration modestly improves performance but does not close the performance gap.
Response to Reviewer NYUB (1 / N)
We thank the reviewer for recognizing the value of our benchmark and analysis. We address the reviewer’s concerns and questions below.
W1.1. Removing anomaly-type hints in the evaluation setting. (Also part of Q1)
The main reason we provide anomaly-type hints for agents in the main evaluation is that detailed anomaly description helps define what an actual bug is for agents, which, without such guidance, agents may not have a clear understanding of what an anomaly is, putting unnecessary challenges for them. In our initial exploration, we also find that removing these type hints leads to large performance drops for these LLMs. Specifically, their bug identification rate are much worse (detailed results in this response) and, more importantly, the false positives are significantly higher (detailed results in the next response). To provide more evidence, beyond the ablation in Table 3, we now report results across all 213 tasks for the three strongest backbones after removing anomaly-type hints in the following table.
As can be seen in this result table, all models perform significantly worse than with anomaly type hints in the context. Nonetheless, we agree with the reviewer that the ability to identify bugs without type hints is necessary for auditing agents. As future work, we plan to make the evaluation more challenging as models and agents become more capable, for example by removing these hints and increasing scene size.
W1.2. Evaluating false-positive behavior on normal scenes. (Also part of Q1)
We agree with the reviewer that false-positive evaluation is necessary for our auditing agent. In addition to the previous response, we evaluate agents on 27 baseline scenes without injected target anomalies (21 Unreal Engine and 6 Three.js scenes) and report false-positive rates. In this experiment, each scene is tested from five different spawn positions, yielding 135 evaluation cases. The positive-report rates are shown below:
These high false positive rates show that reliably recognizing normal scenes remains challenging for current models. Models with higher success rates on anomaly tasks tend to exhibit lower false-positive rates on anomaly-free scenes.