- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces PlayWorld, a benchmark for evaluating interactive video world models on long-horizon objectives.
The main idea is that using the same fixed action sequence may not be fair for different world models, since they can respond differently to the same actions. To address this, the authors use an Agent Player (a multi-modal agent simulates a human player) to adjust the actions based on the generated frames, so that different models can be evaluated on the same objective.
The benchmark contains 171 scenarios and evaluates world models from four aspects: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. The authors evaluate nine world models and show that current models still struggle with long-term consistency and realistic interactions.
Critical strengths
I think the motivation is reasonable. Different world models may have different action speeds and control behaviors, so evaluating them with fixed action sequences may not always be fair. The paper is generally well organized, and the figures provide good examples of the problems that current world models have.
Using an Agent Player to adapt actions is an interesting idea. It makes the evaluation closer to how a human would actually interact with a world model. (Though itself mainly relies on an existing multimodal LLM, so, I see the main contribution as the evaluation protocol rather than a new agent architecture)
The benchmark is relatively comprehensive, with 171 scenarios and evaluations on 9 world models, including both open-source and closed-source models.
I would also appreciate the human evaluation. The authors compare the VQA scores with human preferences and report relatively high correlations.
Critical weaknesses
- Does the Agent Player really make the evaluation fairer?
This is my main concern. The paper argues that fixed action sequences are not fair across different world models, and proposes using an Agent Player to solve this problem. I understand why the authors use adaptive actions, since different world models may respond differently to the same controls. However, this also means that models may be evaluated with different action sequences and rollout lengths. I wonder if these differences introduce additional bias into the final scores. Table 4 shows better trajectory completion, but does not directly demonstrate that the resulting model comparison is fairer.
In Table 4, the authors show that Preset + Agent achieves better trajectory scores and human preferences than Preset Only. However, I think this mainly shows that the Agent Player can better reach the intended objective. It does not necessarily show that the final evaluation scores are more reliable or fair across models.
I would like to see a more direct comparison between fixed-action and agent-based evaluation if possible.
- The evaluation protocol is not exactly the same for all models.
The authors mention that some cases are skipped for HY-World2 because of its input requirements. Hunyuan-GameCraft also cannot properly handle some WAIT-only cases. I understand that different world models have different interfaces and limitations. However, if some models are evaluated on fewer or different cases, I wonder how fair the overall comparison is. It would be helpful to report the number of valid cases for each model and compare the rankings on a common subset of cases.
Also, since Overall is an unweighted mean of the four dimensions (Table 2), I feel like a different case subset changes it directly.
- More analysis of the VQA evaluation would be useful.
The authors use Gemini 3.1 Pro to evaluate the generated videos. I wonder how reliable the VQA scores are for individual cases, e.g. some questions about physical interactions or out-of-sight evolution may be harder for a VLM to judge than simple visual consistency questions.
Questions for authors
Please refer to weakness section.
2 comments
Official Comment by Authors
Dear Reviewer qs22,
We would like to kindly follow up on our rebuttal. We would appreciate it if you could take a look at our responses and let us know if you have any remaining questions or concerns.
Thank you for your time and consideration.
Best regards,
Authors
Response to reviewer qs22
Thank you for your valuable time in reviewing our paper.
The Agent Player makes evaluation fairer by ensuring all models complete the required trajectory, and moderate differences in rollout duration do not bias VQA scores. In Out-of-sight Evolution tasks, the target object must leave the view and then return. Under fixed action sequences, models with smaller step sizes never move the object out of view, which unfairly rewards static models with high consistency scores. Instead, our Agent Player simulates a human player end-to-end to obtain valid trajectories, as shown in Figure 3 and Figure 4. We have also provided more comparison cases on our anonymous project website in the Appendix: https://playworld-2026.github.io/project/PlayWorld/. We also tested shortening video durations by increasing the frame rate, which produced nearly identical VQA scores (mean difference 0.03), confirming that rollout duration differences do not bias the scores.
Thank you for pointing this out. HY-World2 fails to generate 15 cases because its 3D reconstruction upload filter rejects close-up object and face images, leaving 156/171 generated videos: 45/48 GC, 46/50 IF, 25/30 IE, and 40/43 OE. For GameCraft on WAIT-only IE tasks, we provide zero action speed to keep the camera stationary; however, the model cannot remain completely stationary and exhibits camera drift. We regard this as a limitation of the model's own control capability, so we retain all 30/30 GameCraft IE videos in the evaluation. All other models generate all 171 cases.
For each dimension score, HY-World2's ungenerated cases are excluded from the denominator, and the Overall score is computed as the unweighted average of the four final dimension scores. Because more than 91% of videos are shared across all models, this has little impact on the comparison. Evaluating all models on the shared subset only produces the exact same #1 to #9 ranking as Table 2.
To evaluate each case more accurately, our VQA evaluation relies on human-written, scene-specific questions. Rather than only checking simple visual consistency, we manually write VQA questions for each case that specify both expected physical outcomes and common failure modes, such as whether a character stops at an obstacle or splashes when stepping into water. For out-of-sight evolution, our questions mainly focus on comparing the revisited scene with the initial image to check both what should progress and what should stay unchanged, with concrete case scoring examples shown in Figure 3 and Figure 4. To help the VLM capture both motion and fine details, we feed the verifier 10 FPS temporal grids at 384×216 together with 0.5 FPS spatial detail grids at 800×450, as detailed in Appendix A.