acceptodds
Under review as a conference paper at ICLR 2027

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents

Abstract

As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or objects inconsistent with the surrounding scene. Multimodal AI systems, including vision-language models (VLMs) and vision-language-action models (VLAs), have shown potential for automating this task. However, 3D world auditing is complex, requiring the close coupling of two distinct capabilities: action, to navigate the 3D world and search for anomalies systematically and efficiently; and visual reasoning, to understand the environment and identify anomalies from multimodal observations. It remains largely unexplored whether multimodal agents can effectively couple these two capabilities—using visual reasoning to identify potential anomalies while taking actions to validate them. In this paper, we introduce WorldAuditBench, a benchmark for 3D world auditing comprising 213 anomaly tasks across 13 environments built with Unreal Engine 5 and Three.js, spanning five anomaly families. We evaluate five frontier models under a fixed exploration budget using two auditing paradigms: VLA-based exploration followed by VLM-based anomaly identification, and an end-to-end VLM agent in which visual reasoning directly guides action selection. Across the evaluated models and two paradigms, success rates range from 6.6% to 42.3%, substantially below human performance (83.4%). Through the task of world auditing, WorldAuditBench provides a testbed for studying how multimodal agents couple action and visual reasoning in interactive 3D environments, while highlighting current limitations in their ability to gather and interpret evidence during exploration.

Primary area:
datasets and benchmarks
Keywords:
Multimodal agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NYUBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldAuditBench, a benchmark for evaluating multimodal agents on anomaly detection in interactive 3D environments. It contains 213 tasks across 13 environments implemented in Unreal Engine 5 and Three.js. The tasks cover five anomaly families, including physical, spatial, temporal, and semantic inconsistencies.

The authors evaluate two approaches: an interactive VLM agent that reasons and acts during exploration, and a two-stage VLA–VLM system that explores first and analyzes the recorded trajectory afterward. Five model backbones are evaluated, together with a human baseline. The results show that current agents perform substantially worse than humans, and interactive VLM agents generally outperform the two-stage approach. The paper also analyzes anomaly exposure, task guidance, exploration budgets, and multiple anomalies.

Critical strengths
  1. The benchmark addresses a useful problem that is not well covered by existing embodied AI benchmarks. Some anomalies, such as missing collision or disappearing objects, cannot be reliably identified from a single image. They require interaction or comparison across observations. The inclusion of different anomaly types and interactive environments makes this benchmark potentially useful for evaluating more realistic agent behavior.

  2. The dataset construction and analysis require considerable effort. The authors manually create the anomalies and use independent human validation to check their quality. I also find the analysis of anomaly exposure versus identification useful. It helps distinguish whether an agent fails because it never encounters the anomaly or because it cannot identify the anomaly after observing it. This is more informative than reporting success rates alone. The project page is also well designed, with an interactive demo that makes the benchmark tasks more accessible and easier to understand.

Critical weaknesses
  1. The evaluation setting makes the auditing task considerably easier than general anomaly discovery. Each main evaluation task contains exactly one target anomaly, and agents receive an anomaly-type hint and a matching in-context example. The ablation in Table 3 shows that removing these hints reduces the VLM success rate from 33.3% to 15.9% on Unreal Engine tasks. This suggests that the results depend substantially on knowing what kind of anomaly to search for. There are also no normal environments in the main evaluation, so false positive behavior is not measured. In addition, the evaluation protocol does not consistently require evidence from the agent's trajectory to verify the reported anomaly. Unless explicitly required by the rubric, a semantically matching report may receive full credit without sufficient visual evidence of the claimed anomaly. This makes it harder to determine whether the agent actually observed and verified the anomaly.

  2. The comparison between the two auditing paradigms does not sufficiently isolate the effect of reasoning-guided exploration. In the VLM-only setting, the evaluated VLM backbone directly selects exploration actions, while the VLA–VLM setting uses the same Open-P2P 1.2B model for exploration across all backbones. The two approaches also have different exploration budgets and available reasoning capabilities. Thus, the performance gap cannot be attributed only to coupling reasoning with action. The results in Appendix D further show that the two-stage approach performs better for several backbones on Three.js environments, which suggests a strong dependence on the environment and exploration policy. I think this limits one of the paper's main conclusions.

Post-rebuttal Update

The authors have provided substantial additional experiments addressing my original concerns. These results strengthen the evaluation and clarify several limitations of the original benchmark. Some concerns remain regarding the scope of open-ended auditing and the interpretation of the comparison between auditing paradigms. However, the additional evidence addresses my main methodological concerns.

I have therefore updated my rating from 2 to 3. I appreciate the authors' detailed responses and efforts during the discussion.

Questions for authors
  1. How much would the reported success rates change if agents were evaluated without anomaly-type hints and with normal scenes included? In particular, can the authors report false positive rates and evidence-grounded success rates? The existing guidance ablation is helpful, but it does not fully address this issue.

  2. Can the authors provide a more controlled comparison between online reasoning and offline trajectory analysis, using the same exploration backbone or a matched exploration budget? This would help determine whether the advantage comes from online reasoning or from the different exploration policies.

  3. How often do agents receive a successful score when their trajectories do not contain sufficient evidence to verify the reported anomaly? Would requiring such evidence change the ranking between methods? I understand that some rubrics already impose evidence requirements, but it is unclear how consistently this is enforced across the benchmark.

10 comments
AuthorsOpenReview ↗

Response to Reviewer NYUB (1 / N)

We thank the reviewer for recognizing the value of our benchmark and analysis. We address the reviewer’s concerns and questions below.

W1.1. Removing anomaly-type hints in the evaluation setting. (Also part of Q1)

The main reason we provide anomaly-type hints for agents in the main evaluation is that detailed anomaly description helps define what an actual bug is for agents, which, without such guidance, agents may not have a clear understanding of what an anomaly is, putting unnecessary challenges for them. In our initial exploration, we also find that removing these type hints leads to large performance drops for these LLMs. Specifically, their bug identification rate are much worse (detailed results in this response) and, more importantly, the false positives are significantly higher (detailed results in the next response). To provide more evidence, beyond the ablation in Table 3, we now report results across all 213 tasks for the three strongest backbones after removing anomaly-type hints in the following table.

BackboneVLM(w/ hints → w/o hints)VLM–VLA(w/ hints → w/o hints)
GPT-6 Astra42.3% → 24.9%16.4% → 11.7%
Gemini 3.8 Flash32.4% → 18.8%17.4% → 7.5%
Claude Opus 528.2% → 14.1%15.0% → 8.0%

As can be seen in this result table, all models perform significantly worse than with anomaly type hints in the context. Nonetheless, we agree with the reviewer that the ability to identify bugs without type hints is necessary for auditing agents. As future work, we plan to make the evaluation more challenging as models and agents become more capable, for example by removing these hints and increasing scene size.

W1.2. Evaluating false-positive behavior on normal scenes. (Also part of Q1)

We agree with the reviewer that false-positive evaluation is necessary for our auditing agent. In addition to the previous response, we evaluate agents on 27 baseline scenes without injected target anomalies (21 Unreal Engine and 6 Three.js scenes) and report false-positive rates. In this experiment, each scene is tested from five different spawn positions, yielding 135 evaluation cases. The positive-report rates are shown below:

BackboneVLMVLM–VLA
GPT-6 Astra33.3%25.2%
Gemini 3.8 Flash60.0%63.0%
Claude Opus 588.9%77.0%

These high false positive rates show that reliably recognizing normal scenes remains challenging for current models. Models with higher success rates on anomaly tasks tend to exhibit lower false-positive rates on anomaly-free scenes.

AuthorsOpenReview ↗

Response to Reviewer NYUB (2 / N)

W1.3. Evidence-grounded evaluation. (Also part of Q1 and Q3)

We conduct an additional evidence-grounded evaluation using an updated judge with stricter visual evidence requirements. Specifically, we modify the evaluation protocol in Appendix C.4 in three ways:

  1. Mandatory visual evidence: A semantically correct report alone is no longer sufficient for a successful score. The reported anomaly must also be supported by the submitted visual evidence.
  2. Evidence sufficiency: The evidence must demonstrate the reported inconsistency. For anomalies involving changes or physical interactions, the submitted evidence must contain the necessary observations to verify the anomaly, such as before- and after-event frames.
  3. Stricter scoring: Reports with missing, irrelevant, or insufficient visual evidence receive a score of 0, even if their textual descriptions match the target anomaly.

We also report human eval results to show that our updated judge aligns with human judgment.

BackboneMethodOriginal scoreEvidence-grounded score (by model)Evidence-grounded score (by human)
GPT-6 AstraVLM42.3%42.3%42.3%
GPT-6 AstraVLM–VLA16.4%14.6%16.4%
Gemini 3.8 FlashVLM32.4%31.0%31.9%
Gemini 3.8 FlashVLM–VLA17.4%15.5%16.4%
Claude Opus 5VLM28.2%27.7%27.7%
Claude Opus 5VLM–VLA15.0%13.6%14.6%

Human assessment reduces success rates by 0.0–1.0 percentage points, while model-based assessment reduces them by 0.0–1.8 percentage points. These results suggest that receiving a successful score without sufficient supporting evidence in the observed trajectory is uncommon. In our human review, the few errors mainly concern evidence selection: omitting a necessary before- or after-event frame, or including irrelevant frames, which we count as errors. When a necessary frame is omitted from the selected evidence, it is still actually available in the full trajectory.

AuthorsOpenReview ↗

Response to Reviewer NYUB (3 / N)

W2. Fairness of the comparison between auditing methods. (Also Q2)

We examine comparison fairness along three dimensions: exploration distance, model backbones, and inference costs.

The main evaluation uses 40 actions for the VLM and a 60-second exploration setting for the VLA, aiming for roughly comparable exploration distances. We will clarify this comparison in the revised PDF.

To address the remaining two dimensions, we additionally conduct matched-backbone experiments and an inference-cost budget analysis, as detailed below.

Below is the result for the matched-backbone experiments:

SettingGPT SR
A. VLM agent (GPT)42.3%
B. GPT free exploration → GPT analysis28.6%
C. GPT anomaly-aware exploration → GPT analysis32.4%
D. VLA exploration → GPT analysis16.4%

Each entry covers all 213 tasks. B and C share the same exploration harness, tools, and 40-action budget, differing only in the exploration objective. Both separate exploration and analysis into two roles: an explorer collects observations, and a fresh GPT analyzer reviews the resulting trajectory with the original task context.

B and C outperform D, suggesting a capability gap between the current VLA explorer and GPT-based exploration. A further outperforms B and C, consistent with the benefit of keeping analysis and exploration coupled: its auditor can gather additional evidence as new hypotheses emerge, whereas the separate analyzers in B and C can only review the completed trajectories.

We also aim to control for inference cost. Settings A, B, and C already incur similar costs, whereas D is substantially cheaper. We therefore increase D’s budget through longer exploration recordings and additional offline review rounds, aiming to bring its inference cost closer to that of A–C.

SettingSRMean inference cost/task
A. VLM agent (GPT)42.3%$2.955
B. GPT free exploration → GPT analysis28.6%$2.814
C. GPT anomaly-aware exploration → GPT analysis32.4%$3.015
D1. VLA exploration → GPT analysis (non cost-matching)16.4%$0.281
D2. VLA exploration → GPT analysis (cost-matching)17.4%$3.681

A outperforms B and C at similar inference costs. For D, longer exploration modestly improves performance but does not close the performance gap.

Reviewer_VEAMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a 3D interactive environment for anomaly detection. The benchmark requires multimodal agents to interact with the 3D world and report anomalies. The paper also evaluates two types of auditors and several frontier MLLMs on the proposed WorldAuditBench. The results reveal a critical limitation in the 3D understanding capabilities of current MLLMs.

Critical strengths

The benchmark supports interaction and includes diverse scenes. It also covers a wide range of anomaly types, providing a comprehensive evaluation environment.

The benchmark extends existing anomaly detection benchmarks and explores two critical MLLM capabilities: action and spatial reasoning. This may suggest potential directions for future research.

Critical weaknesses

The experiments and analyses appear insufficient to fully assess MLLM capabilities. First, Figure 6 and Table 3 show that the human-rated exposure rate is below 70% and that reducing the step budget by 50% decreases the success rate (SR) by 10%. These findings may suggest that the models require more steps to explore the 3D world. Moreover, increasing the step budget gives agents more opportunities to encounter anomalies, particularly those involving temporal consistency and interactive physics, which may require additional steps to detect. Second, other factors, such as environmental complexity and scene size, could also affect agent performance. These factors are not evaluated, limiting our understanding of why agents fail in certain cases and what accounts for performance differences among frontier models.

The manuscript provides only one complete report, on page 24, and it is unclear whether the demonstrations on the project website were produced by humans or MLLMs. Additional examples showing how MLLMs interact with the 3D world would help readers analyse their behavior.

Questions for authors

Could the authors provide more details about the benchmark, particularly factors that might affect the success rate? How does the benchmark construction support the evaluation of MLLMs’ action and visual reasoning capabilities? Which aspects of these capabilities does each of the five anomaly categories assess?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VEAM (1 / N)

We thank the reviewer for recognizing the diversity and comprehensiveness of our benchmark, as well as its contributions to evaluating MLLMs' action and spatial reasoning capabilities. We address the reviewer’s concerns and questions below.

W1.1: Exploration Budget and Anomaly Exposure

We thank the reviewer for highlighting the role of exploration budget. We extend the exploration-budget ablation in Table 3 of the original paper by further examining anomaly exposure alongside the already reported auditing success rates. Specifically, we compare Gemini 3.8 Flash under 40-action (1×) and 60-action (1.5×) budgets on the same 126 Unreal tasks. Human-rated anomaly exposure measures whether the recorded trajectories contain sufficient visual or interaction evidence to identify the target anomaly, helping us better understand how exploration budget affects auditing performance.

Anomaly CategoryTasksSuccess Rate (%) (1× → 1.5×)Human-Rated Exposure (%) (1× → 1.5×)
Static Physics3151.6 → 71.071.0 → 90.3
Interactive Physics2218.2 → 18.254.5 → 63.6
Spatial Consistency2722.2 → 14.855.6 → 66.7
Temporal Consistency2619.2 → 15.442.3 → 50.0
Semantic Consistency2055.0 → 50.075.0 → 85.0
Overall12633.3 → 34.959.5 → 71.4

Additional exploration increases exposure across all five categories, raising overall exposure from 59.5% to 71.4%. However, overall SR improves only modestly, from 33.3% to 34.9%. Static Physics benefits substantially in both exposure and success, whereas Interactive Physics SR remains unchanged and the three consistency categories decline despite higher exposure. These results suggest that increasing the exploration budget improves anomaly exposure, but does not necessarily translate into higher auditing success rates. Even when anomalies are successfully exposed, models may fail to recognize them or compare evidence across multiple observations, particularly for tasks requiring spatial or temporal reasoning. This highlights that anomaly recognition and evidence integration, rather than exploration alone, remain key bottlenecks in auditing performance.

AuthorsOpenReview ↗

Response to Reviewer VEAM (2 / N)

W1.2: Impact of Scene Size on Auditing Performance

We compare Gemini 3.8 Flash in regular-size (1×) and smaller (0.5×) Unreal environments, using the same 126 tasks and keeping the other auditing setup unchanged.

Scene Size# TasksSuccess Rate (%)Human-Rated Exposure (%)
Small (0.5×)12650.876.2
Regular (1×)12633.359.5

Reducing scene size increases human-rated exposure from 59.5% to 76.2% and SR from 33.3% to 50.8%. These results indicate that scene size affects auditing difficulty, with smaller environments facilitating both anomaly exposure and successful identification. Consistently, our existing starting-distance ablation on the same tasks shows that moving the starting position closer to the anomaly increases SR from 33.3% to 38.1% (Table 3). Together, these findings highlight spatial accessibility as an important factor in auditing performance.

W2: Demos and More examples

All demonstrations on our project website show trajectories generated by MLLM agents. We will also add more examples and complete auditing reports to the revised PDF soon. We will also release all agent trajectories to support further analysis.

Q: Capability Breakdown Across Anomaly Categories.

We thank the reviewer for this question. All five anomaly categories involve multiple capabilities, including perception, reasoning, and interaction. However, each category places different emphasis on specific capabilities. The table below summarizes the primary capabilities assessed by each category.

Anomaly CategoryVisual PerceptionSpatial ReasoningSemantic ReasoningAction / InteractionCross-Observation Reasoning
Static Physics✓✓
Interactive Physics✓✓✓
Spatial Consistency✓✓✓
Temporal Consistency✓✓
Semantic Consistency✓✓

While static physics and spatial consistency primarily emphasize visual perception and spatial reasoning, interactive physics additionally requires action-based exploration. Temporal consistency emphasizes reasoning across observations and scene states, whereas semantic consistency focuses on understanding object properties and scene-level semantics. Together, these categories provide complementary perspectives on the capabilities required for auditing interactive 3D environments.

The rebuttal has provided more results. I'll maintain my score.

Reviewer_33uFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldAuditBench, an interactive benchmark for evaluating multimodal agents in 3D world anomaly auditing. It contains 213 tasks across 13 environments and five anomaly families, including physical, spatial, temporal, and semantic inconsistencies. The authors evaluate two paradigms: VLM agents that jointly reason and act during exploration, and VLA–VLM agents that perform exploration followed by offline anomaly identification. Experiments demonstrate a substantial gap between current multimodal agents and human performance. Further analyses investigate anomaly exposure, recognition, exploration budgets, and the effects of task guidance.

Critical strengths

The paper introduces an interesting evaluation task that requires multimodal agents to actively explore 3D environments and gather evidence for anomaly verification. Unlike conventional image- or video-based anomaly detection benchmarks, WorldAuditBench emphasizes the interaction between visual reasoning and action, making it relevant to embodied intelligence research.

Critical weaknesses
  1. The VLM and VLA–VLM approaches differ in several important aspects, including exploration models, action budgets, memory tools, and computational resources. Consequently, the substantial performance gap cannot be attributed solely to the coupling of visual reasoning and action. More controlled comparisons are needed to isolate the contribution of interactive reasoning from differences in exploration capability and tool availability.
  2. The benchmark provides agents with anomaly-type hints and corresponding in-context examples, which substantially simplify the anomaly discovery task. As shown in Table 3, removing both forms of guidance reduces the VLM success rate from 33.3% to 15.9%. This raises concerns about whether the benchmark adequately evaluates open-ended anomaly discovery rather than guided anomaly search. The paper would benefit from more systematic evaluation under guidance-free settings.
Questions for authors
  1. Can the authors provide more controlled comparisons between VLM and VLA–VLM auditors under matched exploration budgets and tool access? How much of the observed performance difference can be specifically attributed to reasoning-guided action selection?
  2. Given the substantial performance degradation without anomaly-type hints, how well does WorldAuditBench evaluate open-ended anomaly discovery? Can the authors provide more comprehensive results without predefined anomaly guidance?
  3. How robust are the benchmark results to the choice of evaluation judge? Since GPT-6 Astra is used both as an evaluated agent and as the evaluation judge, can the authors provide independent human evaluations or cross-judge agreement analysis for reports generated by different models?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 33uF (1 / N)

We thank the reviewer for recognizing the significance of our benchmark and its emphasis on integrating visual reasoning with interactive exploration for anomaly detection in 3D environments. We address the reviewer’s concerns and questions below.

W1. Fairness of the Comparison Between Auditing Methods (Also Q1)

We thank the reviewer for raising this concern. We examine comparison fairness along three dimensions: exploration distance, model backbones and tool access, and inference costs.

1. Exploration Distance. The main evaluation uses 40 actions for the VLM and a 60-second exploration setting for the VLA, aiming for roughly comparable exploration distances. We will clarify this comparison in the revised paper.

2. Model Backbones and Tool Access. The original VLA explorer (Open-P2P) is a low-level action policy without native support for high-level reasoning, memory, or structured tool use. Directly matching its tool access with the VLM auditor would require fundamentally modifying the VLA architecture. To address this limitation, we introduce matched-backbone experiments using GPT-6 Astra for both exploration and analysis, enabling more controlled comparisons.

SettingGPT SR (%)
A. VLM agent (GPT)42.3
B. GPT free exploration → GPT analysis28.6
C. GPT anomaly-aware exploration → GPT analysis32.4
D. VLA exploration → GPT analysis16.4

Each setting covers all 213 tasks. B and C share the same exploration backbone, harness, tools, and 40-action budget, differing only in their exploration objectives. Both separate exploration and analysis into two stages: an explorer collects observations, and a fresh GPT analyzer reviews the resulting trajectory with the original task context.

B and C outperform D, suggesting that the exploration backbone's capabilities contribute to the original performance gap. Furthermore, C outperforms B (32.4% vs. 28.6%), demonstrating the benefit of anomaly-aware exploration under matched backbone and tool configurations.

A further outperforms B and C (42.3% vs. 28.6%/32.4%), consistent with the benefits of coupling reasoning and exploration. In A, the auditor can continuously formulate hypotheses and gather additional evidence as new observations emerge, whereas the separate analyzers in B and C can only review completed trajectories. These discoveries also motivates our ongoing follow-up work on training exploration models with integrated reasoning capabilities.

3. Inference Costs. Settings A, B, and C already incur similar inference costs, whereas D is substantially cheaper. We therefore increase D's computational budget through longer exploration recordings and additional offline review rounds to examine whether the performance gap can be explained by inference cost differences.

SettingSR (%)Mean Inference Cost / Task
A. VLM agent (GPT)42.3$2.955
B. GPT free exploration → GPT analysis28.6$2.814
C. GPT anomaly-aware exploration → GPT analysis32.4$3.015
D1. VLA exploration → GPT analysis (non cost-matching)16.4$0.281
D2. VLA exploration → GPT analysis (cost-matching)17.4$3.681

A outperforms B and C at similar inference costs. For D, longer exploration and additional offline analysis modestly improve performance but do not close the performance gap, even at a higher inference cost.

Together, these experiments provide a more controlled analysis of the contributions of exploration capability, anomaly-aware action selection, and integrated auditing.

AuthorsOpenReview ↗

Response to Reviewer 33uF (2 / N)

W2. Evaluation Without Anomaly-Type Hints (Also Q2)

We thank the reviewer for raising this concern. The main reason we provide anomaly-type hints for agents in the main evaluation is that detailed anomaly description helps define what an actual bug is for agents, which, without such guidance, agents may not have a clear understanding of what an anomaly is, putting unnecessary challenges for them. In our initial exploration, we also find that removing these type hints leads to large performance drops for these LLMs.

To further assess performance without predefined anomaly guidance, we extend the ablation in Table 3 to all 213 tasks across the three strongest model backbones, evaluating both auditing paradigms without anomaly-type hints.

BackboneVLM(w/ hints → w/o hints)VLM–VLA(w/ hints → w/o hints)
GPT-6 Astra42.3% → 24.9%16.4% → 11.7%
Gemini 3.8 Flash32.4% → 18.8%17.4% → 7.5%
Claude Opus 528.2% → 14.1%15.0% → 8.0%

As expected, removing anomaly-type hints substantially reduces performance across all models, highlighting the difficulty of open-ended anomaly discovery. These results further motivate our choice of a guided setting for the main benchmark evaluation, while demonstrating its applicability to more challenging guidance-free scenarios. We will clarify this design rationale and include the expanded evaluation in the revised paper.

As future work, we plan to make the evaluation more challenging as models and agents become more capable, for example by removing these hints and increasing scene size.

AuthorsOpenReview ↗

Response to Reviewer 33uF (3 / N)

Q3. Robustness to the evaluation judge.

Human evaluation. As reported in Table 7 of the original paper, we have already validated the GPT-6 Astra judge against independent human judgments on human-baseline reports. The judge achieves 91.72% overall agreement with human evaluations, with a Cohen's κ of 0.645. The agreement remains consistent across Unreal Engine and Three.js environments.

Evaluation SubsetHuman–GPT Judge Agreement (%)Cohen's κ
Overall91.720.645
Unreal Engine92.860.644
Three.js90.000.642

Cross-judge evaluation. To further examine judge sensitivity on model-generated reports, we compare success rates under the original GPT judge and the independent Gemini judge.

Agent modelAuditing methodGPT judge SRGemini judge SR
GPT-6 AstraVLM42.3%45.2%
GPT-6 AstraVLM–VLA16.4%17.8%
Claude Opus 5VLM28.2%30.0%
Claude Opus 5VLM–VLA15.0%17.8%
Gemini 3.8 FlashVLM32.4%33.3%
Gemini 3.8 FlashVLM–VLA17.4%19.7%
Muse Spark 1.3VLM15.0%17.3%
Muse Spark 1.3VLM–VLA6.6%7.5%
Qwen 3.8 FlashVLM8.5%7.5%
Qwen 3.8 FlashVLM–VLA12.7%15.0%

The two judges produce highly consistent results, with differences within 3 percentage points across all settings. The model rankings and relative performance trends remain unchanged, suggesting that our findings are robust to the choice of judge.

Thanks for your careful feedback. I'll maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.