acceptodds
Under review as a conference paper at ICLR 2027

PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Abstract

Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce **PlayWorld**, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: **geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution**. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are provided in the anonymous supplementary material. More results are available on our [project page](https://playworld-2026.github.io/project/PlayWorld/).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video World Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qs22OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PlayWorld, a benchmark for evaluating interactive video world models on long-horizon objectives.

The main idea is that using the same fixed action sequence may not be fair for different world models, since they can respond differently to the same actions. To address this, the authors use an Agent Player (a multi-modal agent simulates a human player) to adjust the actions based on the generated frames, so that different models can be evaluated on the same objective.

The benchmark contains 171 scenarios and evaluates world models from four aspects: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. The authors evaluate nine world models and show that current models still struggle with long-term consistency and realistic interactions.

Critical strengths

I think the motivation is reasonable. Different world models may have different action speeds and control behaviors, so evaluating them with fixed action sequences may not always be fair. The paper is generally well organized, and the figures provide good examples of the problems that current world models have.

Using an Agent Player to adapt actions is an interesting idea. It makes the evaluation closer to how a human would actually interact with a world model. (Though itself mainly relies on an existing multimodal LLM, so, I see the main contribution as the evaluation protocol rather than a new agent architecture)

The benchmark is relatively comprehensive, with 171 scenarios and evaluations on 9 world models, including both open-source and closed-source models.

I would also appreciate the human evaluation. The authors compare the VQA scores with human preferences and report relatively high correlations.

Critical weaknesses
  1. Does the Agent Player really make the evaluation fairer?

This is my main concern. The paper argues that fixed action sequences are not fair across different world models, and proposes using an Agent Player to solve this problem. I understand why the authors use adaptive actions, since different world models may respond differently to the same controls. However, this also means that models may be evaluated with different action sequences and rollout lengths. I wonder if these differences introduce additional bias into the final scores. Table 4 shows better trajectory completion, but does not directly demonstrate that the resulting model comparison is fairer.

In Table 4, the authors show that Preset + Agent achieves better trajectory scores and human preferences than Preset Only. However, I think this mainly shows that the Agent Player can better reach the intended objective. It does not necessarily show that the final evaluation scores are more reliable or fair across models.

I would like to see a more direct comparison between fixed-action and agent-based evaluation if possible.

  1. The evaluation protocol is not exactly the same for all models.

The authors mention that some cases are skipped for HY-World2 because of its input requirements. Hunyuan-GameCraft also cannot properly handle some WAIT-only cases. I understand that different world models have different interfaces and limitations. However, if some models are evaluated on fewer or different cases, I wonder how fair the overall comparison is. It would be helpful to report the number of valid cases for each model and compare the rankings on a common subset of cases.

Also, since Overall is an unweighted mean of the four dimensions (Table 2), I feel like a different case subset changes it directly.

  1. More analysis of the VQA evaluation would be useful.

The authors use Gemini 3.1 Pro to evaluate the generated videos. I wonder how reliable the VQA scores are for individual cases, e.g. some questions about physical interactions or out-of-sight evolution may be harder for a VLM to judge than simple visual consistency questions.

Questions for authors

Please refer to weakness section.

2 comments
AuthorsOpenReview ↗

Response to reviewer qs22

Thank you for your valuable time in reviewing our paper.

W1: Does the Agent Player really make the evaluation fairer? ... adaptive actions also means that models may be evaluated with different action sequences and rollout lengths. I wonder if these differences introduce additional bias into the final scores... I would like to see a more direct comparison between fixed-action and agent-based evaluation if possible.

The Agent Player makes evaluation fairer by ensuring all models complete the required trajectory, and moderate differences in rollout duration do not bias VQA scores. In Out-of-sight Evolution tasks, the target object must leave the view and then return. Under fixed action sequences, models with smaller step sizes never move the object out of view, which unfairly rewards static models with high consistency scores. Instead, our Agent Player simulates a human player end-to-end to obtain valid trajectories, as shown in Figure 3 and Figure 4. We have also provided more comparison cases on our anonymous project website in the Appendix: https://playworld-2026.github.io/project/PlayWorld/. We also tested shortening video durations by increasing the frame rate, which produced nearly identical VQA scores (mean difference 0.030.03), confirming that rollout duration differences do not bias the scores.

W2: The evaluation protocol is not exactly the same for all models. The authors mention that some cases are skipped for HY-World2 because of its input requirements. Hunyuan-GameCraft also cannot properly handle some WAIT-only cases... It would be helpful to report the number of valid cases for each model and compare the rankings on a common subset of cases.

Thank you for pointing this out. HY-World2 fails to generate 15 cases because its 3D reconstruction upload filter rejects close-up object and face images, leaving 156/171 generated videos: 45/48 GC, 46/50 IF, 25/30 IE, and 40/43 OE. For GameCraft on WAIT-only IE tasks, we provide zero action speed to keep the camera stationary; however, the model cannot remain completely stationary and exhibits camera drift. We regard this as a limitation of the model's own control capability, so we retain all 30/30 GameCraft IE videos in the evaluation. All other models generate all 171 cases.

For each dimension score, HY-World2's ungenerated cases are excluded from the denominator, and the Overall score is computed as the unweighted average of the four final dimension scores. Because more than 91% of videos are shared across all models, this has little impact on the comparison. Evaluating all models on the shared subset only produces the exact same #1 to #9 ranking as Table 2.

MethodGCIFIEOEOverall
Genie 32.612.381.421.752.04 (#1)
HappyOyster2.502.081.571.481.91 (#2)
LingBot-World22.092.052.021.171.83 (#3)
LingBot-World2.172.311.381.301.79 (#4)
HY-World22.142.061.131.091.61 (#5)
SANA-WM1.741.951.161.111.49 (#6)
GameCraft1.601.521.091.301.38 (#7)
HY-WorldPlay1.131.561.021.081.20 (#8)
Matrix-Game-3.01.291.251.001.001.14 (#9)

W3: More analysis of the VQA evaluation would be useful. The authors use Gemini 3.1 Pro to evaluate the generated videos. I wonder how reliable the VQA scores are for individual cases, e.g. some questions about physical interactions or out-of-sight evolution may be harder for a VLM to judge than simple visual consistency questions.

To evaluate each case more accurately, our VQA evaluation relies on human-written, scene-specific questions. Rather than only checking simple visual consistency, we manually write VQA questions for each case that specify both expected physical outcomes and common failure modes, such as whether a character stops at an obstacle or splashes when stepping into water. For out-of-sight evolution, our questions mainly focus on comparing the revisited scene with the initial image to check both what should progress and what should stay unchanged, with concrete case scoring examples shown in Figure 3 and Figure 4. To help the VLM capture both motion and fine details, we feed the verifier 10 FPS temporal grids at 384×216384 \times 216 together with 0.5 FPS spatial detail grids at 800×450800 \times 450, as detailed in Appendix A.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer qs22,

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

Best regards,

Authors

Reviewer_bEkWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that fixed, predefined action trajectories are an unfair way to compare interactive video world models, because models differ in action granularity and response dynamics (the same three "turn" commands may complete a 360° rotation in one model and only a partial turn in another). The authors propose PlayWorld, an evaluation paradigm in which a multimodal Agent Player pursues a scene-grounded long-horizon objective and adapts low-level execution online (Keep / Stop / Extend / Correct / End decisions) while using a human-written basic action sequence as a shared prior. The benchmark contains 171 human-annotated cases, 50 action patterns, 10–60 s rollouts, and 820+ sample-specific VQA rubric questions scored by Gemini 3.1 Pro along four dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, plus basic video-quality and action-controllability metrics. Nine world models (including web-only Genie 3 and HappyOyster) are evaluated. The main findings are that long-horizon spatial consistency and persistent state evolution remain weak even for the best model, and that basic video quality / controllability do not predict completion of long-horizon objectives. Validation includes a controlled comparison of Preset Only / Agent Only / Preset + Agent, a 600-judgment human preference study (Spearman ρ = 0.93 overall with VQA scores), and a two-pass verifier-variance analysis.

Critical strengths
  1. The Agent Player formulation targets a genuine comparability flaw. Existing world-model benchmarks (WorldScore, WorldMark, WBench, etc.) replay prescribed controls; when action granularity differs across models, measured "geometry consistency" is confounded with trajectory failure. Replacing scripted control with objective-conditioned closed-loop adaptation is the right abstraction for this evaluation problem, and the Preset + Agent vs. Preset Only / Agent Only comparison (Tab. 4) with human preference judgments (39.6% → 65.6% on Genie 3, 40.8% → 67.4% on HappyOyster) provides direct evidence that the design choice matters rather than being cosmetic. This is consequential because the benchmark's entire claim of fair cross-model comparison rests on it, and the paper substantiates that claim.

  2. Breadth and care of the benchmark construction. 171 human-curated scenarios across four capability dimensions, 50 action patterns, web-only closed systems (Genie 3, HappyOyster) evaluated through the user-facing interface, and a separate basic-ability suite make this a substantial and timely testbed. The human-alignment study (600 pairwise judgments, Fleiss κ = 0.434, rank correlations 0.75–0.98) and the two-pass verifier stability check (mean per-model sample variance 0.0112) are more validation than is typical for benchmark papers. The resulting finding — Genie 3 leads but is far from solved, with out-of-sight and insight evolution near the floor — is a credible and useful snapshot of where interactive world models stand.

Critical weaknesses
  1. All scores are mediated by two proprietary LLM components, and verifier robustness is only partially established. Claude Haiku drives the agent and Gemini 3.1 Pro scores every rubric item. The paper checks two-pass variance and correlation with human rankings, which is helpful, but does not check cross-verifier robustness (a different model family re-scoring a subset) and does not state whether the verifier is blinded to model identity. This is consequential because the ranking in Tab. 2 is the paper's main empirical product; if the judge has systematic preferences for particular visual styles, frame rates, or interface-capture qualities — and one evaluated model, Genie 3, comes from the same organization as the verifier — then cross-model conclusions could shift. A blinded subset re-scored by an independent verifier with rank correlation reported would substantially strengthen the central claim.

  2. Validity gating and per-model interface adapters conflate model capability with evaluation-pipeline success, and per-model coverage is not reported. Invalid rollouts are assigned the minimum dimension score of 1, but validity itself depends on the per-model action adapter and, for web models, on browser automation and capture quality. HY-WorldPlay (41.6% validity) and HY-World2 (51.7%) score far below SANA-WM (80.4%), yet a substantial part of that gap may reflect adapter/interface calibration rather than world-model quality. In addition, HY-World2 skips cases whose initial images do not support global scene modeling, and Hunyuan-GameCraft cannot be evaluated on WAIT-only insight cases; because per-model case counts are not reported, the denominators behind Tab. 2 are unclear and the "same benchmark" comparison is less uniform than presented. Reporting rubric scores conditional on validity, per-model coverage counts, and a held-out adapter-calibration check would make the comparison credible.

Questions for authors
  1. Is the VQA verifier blinded to model identity and model-specific visual artifacts? Can you re-score a subset with a non-Gemini verifier and report rank correlation with Tab. 2?
  2. Please report the exact number of evaluated cases per dimension for each model after skipped/excluded cases (especially HY-World2 and Hunyuan-GameCraft), and clarify how the Overall score handles missing cells.
  3. How were the per-model control adapters calibrated? Can you estimate how much of the validity gap between, e.g., HY-WorldPlay and SANA-WM is attributable to adapter/interface differences rather than model dynamics?
  4. Agent decisions (Claude) and rubric scoring (Gemini) are separate models but see the same frames; have you checked for correlated failure modes, e.g., cases where the agent terminates early and the verifier then rewards a truncated rollout?
  5. How sensitive are the rankings to the choice of agent model (Tab. 5 shows Claude Sonnet has the highest trajectory score but lower human preference than Haiku)? Would a stronger agent change the relative standing of any models?
4 comments
AuthorsOpenReview ↗

Response to reviewer bEkW (part 1/3)

Thank you for acknowledging the value of our idea. We address each concern below.

W1. & Q1.: Is the VQA verifier blinded to model identity and model-specific visual artifacts? Can you re-score a subset with a non-Gemini verifier and report rank correlation with Tab. 2?

We recognize this as a critical concern, and our VQA verifier is strictly blinded to model identity and model-specific visual artifacts. Its prompt only receives the scene description, long-horizon objective, and sample-specific Yes/No questions, without any model names, file paths, or execution logs. To remove model-specific visual artifacts such as resolution and frame rate, we convert every video from all 9 models into the exact same frame grid format before scoring (Appendix A). To further confirm this, we also re-scored all evaluated videos using an independent non-Gemini judge, GPT-5.6-sol, as shown below.

MethodGemini 3.1 Pro (GC / IF / IE / OE → Overall)GPT-5.6-sol (GC / IF / IE / OE → Overall)
Genie 32.74 / 2.40 / 1.51 / 1.81 → 2.12 (#1)3.73 / 1.78 / 1.44 / 1.46 → 2.10 (#1)
HappyOyster2.54 / 2.15 / 1.47 / 1.54 → 1.92 (#2)2.99 / 1.62 / 1.49 / 1.46 → 1.89 (#2)
LingBot-World22.04 / 2.13 / 1.95 / 1.16 → 1.82 (#3)2.75 / 1.63 / 1.76 / 1.08 → 1.81 (#3)
LingBot-World2.11 / 2.23 / 1.33 / 1.43 → 1.78 (#4)2.80 / 2.12 / 1.09 / 1.16 → 1.79 (#4)
HY-World22.14 / 2.06 / 1.13 / 1.09 → 1.61 (#5)2.61 / 1.64 / 1.13 / 1.08 → 1.62 (#5)
SANA-WM1.72 / 1.89 / 1.13 / 1.16 → 1.48 (#6)2.66 / 1.54 / 1.10 / 1.00 → 1.58 (#6)
GameCraft1.62 / 1.52 / 1.21 / 1.31 → 1.42 (#7)1.88 / 1.16 / 1.13 / 1.08 → 1.31 (#7)
HY-WorldPlay1.12 / 1.63 / 1.01 / 1.08 → 1.21 (#8)1.34 / 1.35 / 1.00 / 1.00 → 1.17 (#9)
Matrix-Game-3.01.30 / 1.25 / 1.00 / 1.00 → 1.14 (#9)1.85 / 1.02 / 1.00 / 1.00 → 1.22 (#8)

GPT-5.6-sol achieves Spearman’s ρ=0.983\rho = 0.983 and Kendall’s τ=0.944\tau = 0.944 rank correlation with Gemini 3.1 Pro. We use Gemini 3.1 Pro as our primary verifier given its stronger fine-grained visual reading on multi-frame grids.

AuthorsOpenReview ↗

Response to reviewer bEkW (part 2/3)

W2. & Q2.: Please report the exact number of evaluated cases per dimension for each model after skipped/excluded cases (especially HY-World2 and Hunyuan-GameCraft), and clarify how the Overall score handles missing cells.

Thank you for pointing this out. HY-World2 fails to generate 15 cases because its 3D reconstruction upload filter rejects close-up object and face images, leaving 156/171 generated videos: 45/48 GC, 46/50 IF, 25/30 IE, and 40/43 OE. For GameCraft on WAIT-only IE tasks, we provide zero action speed to keep the camera stationary; however, the model cannot remain completely stationary and exhibits camera drift. We regard this as a limitation of the model's own control capability, so we retain all 30/30 GameCraft IE videos in the evaluation. All other models generate all 171 cases.

For each dimension score, HY-World2's ungenerated cases are excluded from the denominator. Because more than 91% of videos are shared across all models, this has little impact on the per-dimension comparison. The Overall score is the unweighted average of the four final dimension scores (each on a 1–5 scale), so differences in the number of cases per dimension have minimal impact. As an additional check, we evaluate all models on the shared subset only and obtain the exact same #1–#9 ranking as Table 2 (ρ=1.000\rho = 1.000).

We also report a Conditional Valid-Only Score, which removes invalid rollouts from the denominator instead of assigning them the minimum score of 1.0. The results show that the main conclusions are robust: Genie 3 and HappyOyster remain the top two models, while HY-WorldPlay and Matrix-Game-3.0 remain the bottom two. We think using the minimum score of 1.0 for trajectory failure cases is more reasonable. Removing trajectory failure cases from the denominator would allow a model to obtain a high score by succeeding only on easier cases, without demonstrating reliable interactive world modeling over the full task distribution. We also provide visual quality metrics separately in Table 6.

MethodShared Subset Score (GC / IF / IE / OE → Overall)Conditional Valid-Only Score (GC / IF / IE / OE → Overall)
Genie 32.61 / 2.38 / 1.42 / 1.75 → 2.04 (#1)3.10 / 2.50 / 1.51 / 1.90 → 2.25 (#1)
HappyOyster2.50 / 2.08 / 1.57 / 1.48 → 1.91 (#2)3.26 / 2.23 / 1.47 / 1.66 → 2.16 (#2)
LingBot-World22.09 / 2.05 / 2.02 / 1.17 → 1.83 (#3)2.75 / 2.33 / 1.95 / 1.18 → 2.05 (#4)
LingBot-World2.17 / 2.31 / 1.38 / 1.30 → 1.79 (#4)2.88 / 2.62 / 1.33 / 1.51 → 2.09 (#3)
HY-World22.14 / 2.06 / 1.13 / 1.09 → 1.61 (#5)3.25 / 2.52 / 1.13 / 1.25 → 2.04 (#5)
SANA-WM1.74 / 1.95 / 1.16 / 1.11 → 1.49 (#6)2.11 / 2.02 / 1.13 / 1.18 → 1.61 (#7)
GameCraft1.60 / 1.52 / 1.09 / 1.30 → 1.38 (#7)3.23 / 1.67 / 1.21 / 1.39 → 1.87 (#6)
HY-WorldPlay1.13 / 1.56 / 1.02 / 1.08 → 1.20 (#8)1.83 / 1.79 / 1.01 / 1.25 → 1.47 (#8)
Matrix-Game-3.01.29 / 1.25 / 1.00 / 1.00 → 1.14 (#9)1.73 / 1.28 / 1.00 / 1.00 → 1.25 (#9)

W2. & Q3.: How were the per-model control adapters calibrated? Can you estimate how much of the validity gap between, e.g., HY-WorldPlay and SANA-WM is attributable to adapter/interface differences rather than model dynamics?

Thank you for this insightful question. We believe that the validity gap between models is primarily attributable to the models' own capabilities rather than differences in control-adapter calibration. We report HY-WorldPlay and SANA-WM 3D control-alignment errors in Appendix Table 6, measuring the translation and rotation errors between VGGT-estimated 3D camera trajectories and the target action commands. The errors for HY-WorldPlay and SANA-WM are nearly identical (0.0760.076 / 47.52∘47.52^\circ vs. 0.0670.067 / 46.19∘46.19^\circ), providing evidence that their action interfaces are similarly calibrated. HY-WorldPlay takes pose-step commands (e.g., w-12), while SANA-WM takes key-frame action sequences (e.g., W); both internally convert these inputs into continuous camera trajectories. Our trajectory validity check only evaluates whether the generated trajectory reaches the high-level task objective. Thus, we attribute the remaining validity gap primarily to model dynamics.

For web-based models, browser automation simply replaces human keypresses after the scene has loaded, introducing minimal interface overhead. Moreover, our VLM judges evaluate semantic VQA questions on standardized frame grids rather than raw pixel quality, so differences introduced by browser capture do not affect the verifier's semantic evaluation. This is also consistent with the strong performance of web-evaluated Genie 3 and HappyOyster, which rank #1 and #2, respectively.

AuthorsOpenReview ↗

Response to reviewer bEkW (part 3/3)

W2. & Q4.: Agent decisions (Claude) and rubric scoring (Gemini) are separate models but see the same frames; have you checked for correlated failure modes, e.g., cases where the agent terminates early and the verifier then rewards a truncated rollout?

The agent player and verifier do not influence each other. The agent player (Claude) makes online, step-by-step decisions based only on the partial frames available as the rollout is generated, whereas the verifier (Gemini) evaluates the complete video after generation has finished against the full task objective. Thus, the agent player is solely optimized to follow the desired trajectory and does not make decisions with respect to the verifier or its scoring behavior.

We further validate this by comparing truncated videos with their corresponding full rollouts. The verifier scores consistently decrease when episodes terminate early, since the required revisit or interaction remains incomplete. Moreover, our trajectory validation gate directly assigns the minimum score of 1.0 whenever a trajectory fails to reach the objective, ensuring that truncated rollouts cannot receive artificially high scores.

W2. & Q5.: How sensitive are the rankings to the choice of agent model (Tab. 5 shows Claude Sonnet has the highest trajectory score but lower human preference than Haiku)? Would a stronger agent change the relative standing of any models?

World-model rankings are insensitive to the choice of agent model. As shown in Tables 4 and 5, the agent modifies only ~12.0% of the basic actions, with minimal variation across agent backbones (12.0% for Claude Haiku, 12.4% for Claude Sonnet, and 12.6% for Gemini 3.1 Pro). This is because the agent’s role is simply to guide the rollout along the expected evaluation trajectory. Both Haiku and Sonnet produce valid videos that satisfy our evaluation requirements, despite minor differences in human preference. For example, when the task requires the agent to walk around a central fountain, a weaker agent may follow a less circular path, but it can still successfully make the fountain go out of sight and then reappear, which is sufficient to satisfy the fair evaluation. Since online action adjustment is a lightweight decision task where low latency is critical, we choose Claude Haiku. When stronger and faster VLM agents become available, our framework can also directly adapt to them.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer bEkW,

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

Best regards,

Authors

Reviewer_tMyTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a "PlayWorld", which is a benchmark to evalaute the geometry consistency, out of sight evoluatinteraction fidelity, and insight evlolution, with 171 scenarios. This paper addresses an important insue for the video world models with interactions, especially on the evaluation side.The results find that the insight evolution for the existing methods are weak which provides valuable information.

Critical strengths

This paper is easy to follow and idea is very clear and interesting. Four aspects of the analysis is very comprehensive. The "PlayWorld" agent idea is interesting. In general, this paper made a good contribution on: New benchmark, metrics and new expansion of datasets. For results and validation: Human and VLM hybrid approach is a viable and scalable way. The finding is insightful and consistent.

Critical weaknesses

The weakness could be the VQA quality, since the judgements is only 600, which is from 5 participants. Only 5 participants may introduce losts of baiscs, which is further reflected by the unanimous agreement ratio is so low. Authors are encouraged to further spell out the novelty of agent player design. For benchmark, reporting existing benchmark video size is also needed.

Questions for authors

Increase the number of judge and participant will be helpful.

Authors are encouraged to further spell out the novelty of agent player design.

Authors are encourage to give the number of videos for existing benchmark in Table 1, which will help readers to better appreciate this work's effort.

2 comments
AuthorsOpenReview ↗

Response to reviewer tMyT

Thank you for the positive assessment. We address each concern below.

W1. & Q1: Increase the number of judge and participant will be helpful.

Thank you for your comments. We expanded our human study from 5 to 50 participants and increased the total judgments from 600 to 2,400, which further confirms strong human agreement and alignment with our VQA verifier. Unanimous agreement is naturally low because every rater must independently pick the exact same option, so even one rater choosing Tie reduces unanimity to zero. Majority agreement reaches 95.8% overall with Fleiss’ κ=0.434\kappa = 0.434 in the 5-rater study, and further improves to 96.7% with Fleiss’ κ=0.462\kappa = 0.462 and VQA overall correlation ρ=0.950\rho = 0.950 with 50 participants.

Human Study#Participants#JudgmentsMajority AgreementFleiss' κ\kappaVQA Overall Correlation ρ\rho
Original Table 7560095.8%0.4340.933
Expanded Study502,40096.7%0.4620.950

W2. & Q2: Authors are encouraged to further spell out the novelty of agent player design.

Our goal is to use an agent to simulate the real experience of a human player interacting with world models and to solve a practical cross-model evaluation problem, rather than to propose a new VLM architecture. We tailored the Agent Player with several designs to better fit our evaluation tasks. For example, we give the agent a human-annotated basic action sequence as a starting plan and only allow small online adjustments at each step, which effectively solves the problem of excessive agent thinking time and trajectory drift when planning from scratch (modifying 14.9% of basic actions while boosting human preference from 24.4% to 67.4% in Table 4). In addition, we designed the agent interface to connect unified keyboard and camera commands to both local open-source models and web-based closed-source models like Genie 3 under a single setup, whereas most existing benchmarks overlook these closed-source SOTA models.

W3. & Q3: For benchmark, reporting existing benchmark video size is also needed.

Thank you for pointing this out. To address this, we added #Scenarios and #Evaluation Questions to Table 1. Our benchmark evaluates interaction cases where an agent observes frames and takes up to 40 adaptive steps per video over 10 to 60 seconds. Building each interactive case is much harder because we must select a navigable initial image, verify a feasible multi-step action plan, and write 4 to 8 sample-specific VQA questions. Existing interactive world-model benchmarks include 32 cases in GameWorld Score, 70 in WorldSimBench, 75 in STEVO-Bench, 120 camera-model cases in Omni-WorldBench, and 158 navigation cases in WBench, among which our 171 interactive cases represent a substantial engineering effort.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer tMyT,

We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.

Thank you for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.