- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Dual Exam, a benchmark for evaluating Multimodal Agents (MAs) and World Models (WMs) under controlled conditions. Motivated by the difficulty of diagnosing failures in coupled MA–WM systems, the framework separately evaluates Agents using fixed environment responses and World Models using fixed action interventions. It covers four domains: physical, digital, social, and scientific. The authors evaluate 10 Agent configurations on 1,088 cases and 16 World Model configurations, with 15 evaluated on the full set of 400 cases. The results reveal substantial limitations in Agent task execution and World Model state consistency, highlighting capability gaps relevant to future MA–WM co-evolution.
Critical strengths
-
Meaningful Motivation and Evaluation Design. Diagnosing component-level limitations in Agent–World Model systems is an important problem. The controlled evaluation design provides a reasonable way to isolate Agent execution capabilities from World Model state-transition capabilities, offering more targeted assessments than end-to-end task success alone.
-
Broad Coverage and Detailed Evaluation. The benchmark covers diverse domains, tasks, and model configurations. Its emphasis on verifiable task outcomes, action effects, and state consistency, supported by detailed failure analyses, provides useful insights into limitations that conventional success metrics or visual quality assessments may overlook.
Critical weaknesses
-
Disconnect Between Motivation and Actual Contribution. The paper motivates its significance through the interdependence between multimodal agents and world models, yet the main experiments do not leverage this interdependence to generate new scientific insights. Instead, the proposed framework largely consists of two independent evaluation tracks. It remains unclear what additional scientific or diagnostic value emerges from combining them, beyond conducting existing types of evaluations separately.
-
Defensive Scope Clarifications Do Not Substitute for Demonstrating Contributions. Throughout the paper, the authors repeatedly emphasize what their framework does not aim to establish, such as causal attribution of coupled failures or actual MA–WM co-evolution, while also acknowledging missing evaluation evidence, including evaluator agreement, statistical uncertainty, negative controls, and reproducibility details. Some of these omissions concern the fundamental validity and reliability of the proposed benchmark, rather than merely its scope. While acknowledging limitations is appropriate, such defensive clarifications should not serve as a substitute for demonstrating the significance of the actual contribution. Narrowing the scope of the claims does not resolve the fundamental question of what meaningful advancement the proposed framework offers over existing approaches.
Questions for authors
-
Can the authors provide evidence that the separate evaluation tracks offer meaningful diagnostic insights into actual coupled Agent–World Model systems, beyond what existing independent benchmarks provide?
-
How was the reliability of the World Model scoring process validated, particularly regarding evaluator agreement, scoring consistency, and simple control baselines?
-
To what extent do differences in execution harnesses, tools, and computational budgets affect the Agent leaderboard rankings, and are the main conclusions consistent under comparable evaluation settings?
Diagnostic evidence from an actual coupled rollout. Thank you for asking what the two-track design adds to the study of coupled systems. In the recorded 14-step GPT-5.6 Sol–Seedance 2.5 packing episode, the task required carrying three items in the same blue backpack. Later frames showed a smaller blue pouch. The audit identified four container-identity discrepancies, and none received a targeted Agent response. The two responses among seven eligible events were both navigation retries. The terminal failure outcome records only whether the task was completed. The specification yields two concrete tests for the same failure prefix: whether the WM preserves container identity when given the recorded action from a verified state, and whether the Agent detects or corrects the discrepancy when given a verified observation. The trace provides retrospective evidence of what happened. The replay interventions would test the causal role of each behavior.
WM scoring reliability. GPT-Sol judged video outputs using the full generated video, the action instruction, and the case-specific rubric, assigning a binary label to each applicable check. Independent rescoring of the same outputs would quantify agreement. Scoring copied frames and model outputs on matched cases would show how much apparent performance comes from persistence and whether the models deliver the requested changes. The submission reports neither measured static control scores nor independent agreement estimates. The packing audit illustrates the kinds of discrepancies the protocol can record. Its labels have yet to be independently adjudicated.
Agent comparisons under comparable settings. A six-configuration subset used the archived Claude Code harness on the same 30 SWE tasks. GPT-5.6 Sol and Qwen 3.8 Max passed 10/30 and 7/30, respectively. The subset supports comparisons on these repair tasks. Across the full inventory, GPT 6 Astra remains first under equal-domain weighting, while several other positions change. Quantifying the effects of harnesses, tools, and budgets on the overall order would require running the same configurations and tasks under matched settings.