acceptodds
Under review as a conference paper at ICLR 2027

Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

Abstract

World Models (WMs) can support Multimodal Agents (MAs) through simulated experience and action-conditioned predictions, while MA interactions can provide feedback for refining WMs. This potential reciprocal relationship motivates assessing both components. A pilot study of one MA–WM pairing across ten scenarios and 150 cases illustrates a diagnostic limitation: coupled task outcomes alone cannot identify the source of failure. We introduce Dual Exam, a unified framework that evaluates the two components separately under controlled conditions, assessing MAs under a fixed environment response mechanism and WMs under fixed action interventions. Across four representative domains including physical, digital, social, and scientific, we evaluate ten MA configurations on 1,088 cases spanning 15 reporting categories and 16 WMs across eight scenarios, with 15 evaluated on the full set of 400 cases. The highest MA Index and WM overall scores are 31.3 and 21.7 on their respective 0–100 scales, revealing substantial capability gaps under the evaluated task requirements. These findings position Dual Exam as a diagnostic framework for assessing capabilities relevant to general-purpose multimodal co-evolution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World model, Multimodal Agents, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vXgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dual Exam, a benchmark for evaluating Multimodal Agents (MAs) and World Models (WMs) under controlled conditions. Motivated by the difficulty of diagnosing failures in coupled MA–WM systems, the framework separately evaluates Agents using fixed environment responses and World Models using fixed action interventions. It covers four domains: physical, digital, social, and scientific. The authors evaluate 10 Agent configurations on 1,088 cases and 16 World Model configurations, with 15 evaluated on the full set of 400 cases. The results reveal substantial limitations in Agent task execution and World Model state consistency, highlighting capability gaps relevant to future MA–WM co-evolution.

Critical strengths
  1. Meaningful Motivation and Evaluation Design. Diagnosing component-level limitations in Agent–World Model systems is an important problem. The controlled evaluation design provides a reasonable way to isolate Agent execution capabilities from World Model state-transition capabilities, offering more targeted assessments than end-to-end task success alone.

  2. Broad Coverage and Detailed Evaluation. The benchmark covers diverse domains, tasks, and model configurations. Its emphasis on verifiable task outcomes, action effects, and state consistency, supported by detailed failure analyses, provides useful insights into limitations that conventional success metrics or visual quality assessments may overlook.

Critical weaknesses
  1. Disconnect Between Motivation and Actual Contribution. The paper motivates its significance through the interdependence between multimodal agents and world models, yet the main experiments do not leverage this interdependence to generate new scientific insights. Instead, the proposed framework largely consists of two independent evaluation tracks. It remains unclear what additional scientific or diagnostic value emerges from combining them, beyond conducting existing types of evaluations separately.

  2. Defensive Scope Clarifications Do Not Substitute for Demonstrating Contributions. Throughout the paper, the authors repeatedly emphasize what their framework does not aim to establish, such as causal attribution of coupled failures or actual MA–WM co-evolution, while also acknowledging missing evaluation evidence, including evaluator agreement, statistical uncertainty, negative controls, and reproducibility details. Some of these omissions concern the fundamental validity and reliability of the proposed benchmark, rather than merely its scope. While acknowledging limitations is appropriate, such defensive clarifications should not serve as a substitute for demonstrating the significance of the actual contribution. Narrowing the scope of the claims does not resolve the fundamental question of what meaningful advancement the proposed framework offers over existing approaches.

Questions for authors
  1. Can the authors provide evidence that the separate evaluation tracks offer meaningful diagnostic insights into actual coupled Agent–World Model systems, beyond what existing independent benchmarks provide?

  2. How was the reliability of the World Model scoring process validated, particularly regarding evaluator agreement, scoring consistency, and simple control baselines?

  3. To what extent do differences in execution harnesses, tools, and computational budgets affect the Agent leaderboard rankings, and are the main conclusions consistent under comparable evaluation settings?

1 comment

Diagnostic evidence from an actual coupled rollout. Thank you for asking what the two-track design adds to the study of coupled systems. In the recorded 14-step GPT-5.6 Sol–Seedance 2.5 packing episode, the task required carrying three items in the same blue backpack. Later frames showed a smaller blue pouch. The audit identified four container-identity discrepancies, and none received a targeted Agent response. The two responses among seven eligible events were both navigation retries. The terminal failure outcome records only whether the task was completed. The specification yields two concrete tests for the same failure prefix: whether the WM preserves container identity when given the recorded action from a verified state, and whether the Agent detects or corrects the discrepancy when given a verified observation. The trace provides retrospective evidence of what happened. The replay interventions would test the causal role of each behavior.

WM scoring reliability. GPT-Sol judged video outputs using the full generated video, the action instruction, and the case-specific rubric, assigning a binary label to each applicable check. Independent rescoring of the same outputs would quantify agreement. Scoring copied frames and model outputs on matched cases would show how much apparent performance comes from persistence and whether the models deliver the requested changes. The submission reports neither measured static control scores nor independent agreement estimates. The packing audit illustrates the kinds of discrepancies the protocol can record. Its labels have yet to be independently adjudicated.

Agent comparisons under comparable settings. A six-configuration subset used the archived Claude Code harness on the same 30 SWE tasks. GPT-5.6 Sol and Qwen 3.8 Max passed 10/30 and 7/30, respectively. The subset supports comparisons on these repair tasks. Across the full inventory, GPT 6 Astra remains first under equal-domain weighting, while several other positions change. Quantifying the effects of harnesses, tools, and budgets on the overall order would require running the same configurations and tasks under matched settings.

Reviewer_gXWKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Dual Exam introduces a framework for evaluating Multimodal Agents (MAs) and World Models (WMs) separately under controlled conditions. It tests Agents with fixed environment responses and World Models with fixed action sequences. The benchmark covers four domains and evaluates 10 Agent configurations and 16 World Model configurations. Results reveal weaknesses in task execution, action following, and state consistency.

Critical strengths

The benchmark covers a wide range of tasks and models. By evaluating Agents and World Models separately, it helps identify their individual weaknesses. The paper also provides detailed failure examples to illustrate the findings.

Critical weaknesses
  1. The added value of the unified framework is unclear. Table 2 shows broader coverage than existing benchmarks, but the Agent and World Model tracks still use different tasks, and their results are reported separately. It would be helpful to explain what new insights Dual Exam provides beyond using existing benchmarks independently.

  2. The leaderboard comparisons are not fully fair or reproducible. Models are evaluated using different harnesses, tools, and execution budgets, and some historical records are incomplete. This makes it difficult to separate model performance from differences in evaluation settings. Appendix A also shows that changing domain weights affects some rankings. More consistent comparisons and complete evaluation records are needed.

  3. The reliability of the scores needs further validation. The paper does not report evaluator agreement, repeated-run results, or confidence intervals. In particular, judging action responses and state consistency in World Models can be subjective. More validation is needed to show that the scores are reliable and reproducible.

Questions for authors

Please refer to the Critical Weaknesses above

1 comment

What the two tracks reveal. Thank you for asking what the joint framework adds. In the WM cup-stacking example, extra cups appear and violate the original quantity constraint. In the coupled packing example, later observations depict a different container and the Agent continues without a targeted identity check. These examples motivate two complementary checks: whether a prescribed action preserves task-relevant state, and whether an Agent verifies an observed state discrepancy before acting on it. Both checks can be formulated from the same specification of the desired change and protected state. Matched replays would quantify how well the checks distinguish the two problems on common episodes.

Comparability and reproducibility. The Agent leaderboard compares model–harness configurations. Six configurations were evaluated on the same 30 SWE tasks with the archived Claude Code harness, with one accepted run per task and configuration. GPT-5.6 Sol passed 10/30, Qwen 3.8 Max 7/30, and Gemini 3.6 Flash 5/30. Seed's 0/30 coincided with failures to enter the task-tool loop on all 30 accepted runs. This example illustrates how execution traces help interpret pass counts. Across the full inventory, equal-domain weighting leaves GPT 6 Astra first and moves DeepSeek V4.1 Flash from sixth to second. The size of the harness and budget effects on the full leaderboard remains unmeasured, and some historical item-level records are incomplete.

WM scoring and coverage. GPT-Sol assigned the binary labels for video outputs using the full generated video, the action instruction, and the case-specific rubric. An independent agreement audit and repeated generations are needed to quantify scoring agreement and run-to-run variability. Separately, we recomputed the means on the six scenarios covered by all configurations. From the displayed scores, Seedance 2.5 averages approximately 18.7, GPT-Image-2 17.6, and MiniMax H3 17.4. The common-scenario comparison addresses the difference in coverage. The paper reports Seedance separately from the eight-scenario ranking.

Reviewer_2U1KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors address the relationship between the World Models (WMs) and the Multimodal Agents (MAs). They claim that the world models can support the multimodel agents as an evaluation under the controlled conditions base on the feedback and this could refine the world model. They provide this in the form of a benchmark with ten configurations on the multimodal agent with fifteen reporting categories and sixteen world model configurations. They scale the values from 0 to 100 as a score and also scored 31.3 and 21.7 for multimodal agent index and the world model scores.

Critical strengths
  1. I think authors clearly separated the main two components they are dealing with.
  • For the agent evaluation, they fixed the environment responses and in the case of evaluation of the world models, they held the behavioral intervention
  • The world model track's evaluation part is well designed. They didn't evaluate the visual quality. Instead, they found out that the requested differences really happened and whether the unrelated part remained without changing.
  1. Wide range of evaluation scope and Clear scope setup
  • Authors evaluated across 4 domains for 16 world models and 10 agent configurations. This seems like a large range of comparison.
  • They also state the limitations honestly so the result looks responsible and it was easy for me to understand the result.
Critical weaknesses
  1. The evaluation scores' reliability is not proven.
  • Based on the appendix B, this paper doesn't present inter-rater agreement and the rate of invalid runs. There is no observation of the confidence intervals of the variance across the repeated runs. Also, lack of the no-op baseline or shuffled baseline. Here, I think the lack of the baseline study raises a big question in the world model track.
  • Various evaluation standards reward state persistence, so the case of the baseline that copies the input frame as an input without any actions could get a higher score on state persistence. Authors should show a lower bound baseline to persuade the world model score is meaningful.
  • The agent index is calculated by adding the item level scores to the mean value of the baselines scores. However, authors indicated that they haven't obtained log for the entire model configuration. This make hard to evaluate whether the order of the models is statistically meaningful.
  1. I think the dual framing is not enough to support This paper's motivation is the multimodel agents and the world model's can improve one another. However, I believe becuase of the following reasons this is logical leap:
  • Two different evaluation track use different task with metrics and don't share common episodes
  • In the pilot experiment, they utilized an individual scenario set.
  • The authors didn't verify the individual component score predicts the performance of the combined system

Therefore, I think this paper suggests two individual benchmarks in one combined name of benchmark, so this could give confusion for the readers. Also, in the case of the agent track some parts combine existing agents' last exam such as SWE-bench or EdgeBench. This limits the contribution and originality of the paper.

Questions for authors

Most of the questions are handled in the Critical Weaknesses part.

Here are some additional questions: 1, For the copied input frames and No-op world model baseline scores across each evaluation metric, respectively?

  • I think if the baseline's calculated score is similar to the evaluated models, then how can we justify the contributions of the paper
  1. How does the evaluation of the binary evaluation criteria of the world model track progressed?
  • I'm curious that does the human annotators annotates direclty, or the llm evaluator works? If so, what model do you use?
1 comment

Thank you for the detailed review. We address the scoring procedure, the controls needed to interpret the WM scores, and the evidence supporting the two-track design.

How the WM outputs were scored. We used GPT-Sol as a vision-language judge for video outputs. For each case, GPT-Sol received the full generated video, the action instruction, and the case-specific rubric. It assigned a binary pass or fail to each applicable rubric check, including checks for the requested effect and preservation of task-relevant state. We will make the judge and its inputs explicit in Section 2.4. The submission currently lacks an independent agreement estimate for these labels, a consolidated invalid-run rate, and confidence intervals from repeated generations.

Agent Index and task provenance. The Index combines historical category means available at one-decimal precision with recorded item-level scores for the added tasks. Incomplete historical item-level logs prevent us from establishing the statistical significance of the full ranking. As a sensitivity check, removing the 30 added SWE-bench repair tasks and ten derived ToolChoice checkpoints leaves GPT 6 Astra first, while Kimi K3 moves from fifth to third. The Scientific set also includes 63 selected Agents' Last Exam tasks and six EdgeBench tasks. We credit these sources and identify ToolChoice as a set of derived checkpoints that tests specific tool decisions within the repair workflows.

Why the tracks are evaluated together. The tracks share a specification structure that states what an action should change, what it should preserve, and what evidence verifies each requirement. In the recorded packing rollout, later observations showed a smaller pouch in place of the required backpack. The Agent made no targeted response to four container-identity discrepancies. That trace motivates two concrete measurements: whether a prescribed action preserves container identity, and whether the Agent responds to a visible identity discrepancy. The pilot used a separate scenario set and recorded zero strict successes across 150 cases for one Agent–WM pair. Testing whether the component scores predict closed-loop performance requires matched evaluations across multiple pairs. Testing reciprocal improvement also requires evaluating updates to the two components over successive rounds.

Reviewer_bbY3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dual Exam, a benchmark framework designed to separately evaluate Multimodal Agents (MAs) and World Models (WMs) under controlled conditions. The central motivation is that the performance of a coupled MA–WM system cannot reveal whether failures originate from agent decisions, world model predictions, or their interaction.

The proposed framework consists of two evaluation tracks. The Agent Examination fixes the environment response mechanism and evaluates agent capabilities in task execution, planning, interaction, and verification. The World Model Examination fixes action interventions and evaluates state transitions, action responsiveness, state persistence, and consistency.

The benchmark covers four domains: physical, digital, social, and scientific. It evaluates ten MA configurations on 1,088 cases across 15 categories, and 16 WM configurations on up to 400 cases across eight scenarios. The reported highest Agent Index and WM Overall scores are 31.3 and 21.7, respectively.

The paper aims to provide a diagnostic foundation for future MA–WM co-evolution research. However, the current work primarily establishes two separate evaluation tracks rather than demonstrating that the proposed framework can reliably characterize or predict the behavior of coupled systems.

Critical strengths

S1. Important and relevant research problem.

The paper identifies a meaningful limitation of evaluating multimodal agents and world models exclusively through end-to-end interaction. When the two components are coupled, an unsuccessful task outcome does not necessarily reveal which component is responsible. Separating their evaluations is a reasonable and potentially useful direction, particularly for systems that rely on generative world models for planning and interaction.

S2. Broad evaluation coverage.

The benchmark covers four different domains, including physical, digital, social, and scientific tasks. The inclusion of heterogeneous task categories is valuable for examining the limitations of current multimodal systems beyond conventional question answering or isolated image and video generation.

The evaluation of ten agent configurations and 16 world model configurations also provides a relatively broad empirical overview of current systems.

S3. Attention to execution evidence and task completion.

The paper emphasizes that successful task completion should be determined by verifiable execution evidence rather than model-generated completion claims. The distinction between locally correct decisions and globally valid task execution is important. Similarly, the WM evaluation considers not only the requested state transition but also whether previously established states and constraints are preserved.

Critical weaknesses

W1. The central methodological contribution is not sufficiently established.

The main contribution is the separation of MA and WM evaluations through fixed environment responses and fixed action interventions. However, this separation is conceptually straightforward, and the paper does not sufficiently establish what methodological advances make Dual Exam substantially different from combining existing agent benchmarks and world model benchmarks with additional evaluation protocols.

The authors compare Dual Exam against several existing benchmarks, but the comparison mainly concerns coverage dimensions. It does not demonstrate that the proposed evaluation design provides substantially more reliable, informative, or actionable measurements than existing evaluation approaches.

In particular, the paper should clarify whether the contribution is a new benchmark collection, a new evaluation methodology, or a unified diagnostic framework. These are different contributions and require different forms of validation.

W2. The relationship between isolated evaluation and coupled system performance remains unvalidated.

The primary motivation is that MA and WM failures interact and compound during coupled execution. However, the proposed framework evaluates the two components independently, and the experiments do not establish whether their isolated scores predict actual coupled performance.

For example, a WM that achieves a higher overall score under fixed interventions may not necessarily perform better on the particular action distribution generated by a given agent. Likewise, an agent that performs well in a fixed environment may behave differently when interacting with a generative WM that introduces state inconsistencies.

The paper acknowledges this issue in the discussion, but it is also a fundamental limitation of the proposed framework. Without matched MA–WM interaction experiments, it remains unclear how useful the two isolated scores are for diagnosing or selecting actual agent–world model combinations.

A more convincing evaluation would include matched agent–WM pairs, measure their closed-loop task performance, and investigate whether the isolated scores correlate with or explain the observed coupled outcomes.

W3. Benchmark aggregation and score comparability are insufficiently justified.

The Agent Index combines 15 categories with substantially different task definitions, difficulty levels, verification procedures, and data sources. The primary aggregation weights categories according to item counts, resulting in a strong influence from physical tasks, particularly navigation and action sequencing.

Similarly, the WM Overall score equally averages eight scenarios despite substantial differences in task complexity, modality, and evaluation criteria.

Although the appendix includes weighting sensitivity analyses, these analyses do not establish that the resulting scores are reliable measures of general MA or WM capabilities.

More importantly, the benchmark combines newly constructed cases, adaptations of existing benchmarks, historical execution records, and externally sourced results. These heterogeneous sources may introduce differences in execution conditions and evaluation reliability that are not reflected in the final aggregate scores.

The authors should provide stronger justification for the aggregation scheme and demonstrate how sensitive the conclusions are to task difficulty, source composition, and evaluation methodology.

Questions for authors

Predictive validity: Have the authors evaluated whether isolated MA and WM scores correlate with the performance of actual coupled MA–WM systems? If not, what evidence supports the claim that Dual Exam provides useful diagnostics for coupled systems?

Evaluation reliability: How are environment responses and task verifiers validated? Can the authors provide inter-annotator agreement, verifier accuracy, or other quantitative reliability measurements?

1 comment

Contribution and predictive validity. Thank you for asking us to clarify what Dual Exam contributes and how it has been validated. The two tracks use a common specification structure that defines the permitted intervention, the required change, the state that must be preserved, and the evidence used to verify each requirement. The Agent track evaluates whether the Agent's chosen actions satisfy the specification under a fixed reference environment response mechanism. The WM track evaluates whether a prescribed action produces its required effect and preserves task-relevant state. This shared structure turns observations from a coupled failure into concrete questions that can be tested under controlled conditions. Table 2 compares benchmark coverage and design. A matched comparison would be needed to quantify any gain in diagnostic accuracy over existing benchmarks.

The pilot evaluated GPT-5.6 Sol with Seedance 2.5 and recorded zero strict successes across 150 scored cases. The single pairing and constant terminal outcome leave the predictive relationship untested. The reviewer's concern about action distributions is reflected in our scenario results. GPT-Image-2 leads MiniMax H3 on WM Overall, 21.7 versus 17.3, while MiniMax H3 leads on Embodiment, 53.8 versus 20.9. This makes evaluation on a candidate pair's own states and actions important when selecting a WM for that Agent.

Aggregation and comparability. The Agent Index weights the evaluated items by count. Physical tasks account for 524 of 1,088 items. Under equal-domain weighting, GPT 6 Astra remains first and DeepSeek V4.1 Flash moves from sixth to second. Recomputing the Index from the reported scores after excluding the 30 newly added SWE repair tasks and ten derived ToolChoice checkpoints leaves Astra first at approximately 30.9 on 1,048 items, while Kimi K3 moves from fifth to third. WM Overall assigns equal weight to eight scenarios with 50 cases each. These aggregates summarize the evaluated task mixtures. The effects of differences in task difficulty, data provenance, and execution conditions across the full inventory remain unquantified.

Verifier reliability. Validation depends on the task family. SWE uses executable tests, while ToolChoice checks the selected tool and required arguments. For WM video outputs, GPT-Sol received the full generated video, the action instruction, and the case-specific rubric, then assigned binary labels to applicable checks. The submission lacks a benchmark-wide verifier accuracy audit and an independent agreement estimate for the WM output labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.