acceptodds
Under review as a conference paper at ICLR 2027

WBench: A Comprehensive Multi-turn Benchmark for Interactive World Model Evaluation

16 views

Abstract

Interactive world models are advancing rapidly, yet existing benchmarks cover only part of the required competencies, leaving no unified standard for systematic evaluation. To fill this gap, we introduce **WBench**, a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions, namely video quality, setting adherence, interaction adherence, consistency, and physics compliance. WBench contains 289 test cases and 1,058 interaction turns, where each case specifies a world setting and a multi-turn interaction sequence, covering diverse scenes, styles, subjects, and both first- and third-person perspectives, together with four interaction types, including navigation, subject action, event editing, and perspective switching. For navigation, WBench unifies text, 6-DoF pose, and discrete-action control, enabling evaluation of models with different native input interfaces. Evaluation uses 22 automatic sub-metrics that combine specialist vision models with vision-language models. Across 20 state-of-the-art models, we find that no single model performs strongly across all dimensions. We hope these diagnostic insights into model strengths, weaknesses, and open challenges will help advance research on interactive world models.

Primary area:
datasets and benchmarks
Keywords:
interactive world model, benchmark, comprehensive, multi-turn

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_mVKuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce WBench, a benchmark for evaluating interactive world models across five complementary dimensions through explicit world settings and multi-turn interactions. In order to do this, the authors use careful design and specifications, and collect data that spans diverse scenarios from multiple levels and perspectives, and I think it basically cover most if not all of the world generation settings. Experiments on 20 representative world models reveal that no model excels in all dimensions. Thus the authors use fine-grained profiles to help identify which capabilities fail under a given interaction, rather than treating visual quality or a single aggregate score as a substitute for world-model competence. This makes sense to me because I do think reporting dimension-level results alongside overall comparisons gives us a more detailed picture as for which model or architecture design works for a certain evaluation dimension or a certain type of interaction.

Critical strengths
  1. A new direction of evaluating interactive world-model. The benchmark goes beyond evaluating single turn modality. The authors propose a benchmark for multi-turn world generation evaluation. The authors do carefully design to support the evaluation through four interaction categories, with both first- and third-person perspectives and multi-turn design. I love the idea the authors proposed that the benchmark score is merged into a single overall one, and different dimension of analysis offer a more comprehensive evaluation on the model.

  2. Multi-dimension evaluation suite with thorough empirical study and metric validation. The benchmark proposes 5 dimensions: video quality, setting, interaction, consistency, physical, with 22 metrics in total. The evaluation covers 20 recent models. The authors also make a substantial effort to validate the benchmark metrics, with 400 annotators and 13,515 pairwise comparisons used for human preference evaluation. These analyses make the benchmark more convincing.

Critical weaknesses
  1. Limited discriminative power of many metrics. I agree with the authors' claim that "No model or paradigm dominates uniformly". However, I notice that several metrics appear highly concentrated or close to saturation across models, which might limits their discrimination ability. For example, Aesthetic score, imaging, flickering, smoothness metrics in video quality dimension, the scores are so close that we can't really safely define which model is better than the other. Also for background, gated spatial, subject, geometric, photometric metrics in consistency dimension, and for visual plausibility metric in physical dimension.

  2. Imbalanced coverage of interaction types. 57% of the dataset is navigation, with only 6% for perspective switching. I wonder if it raises concerns about whether the reported rankings are representative for the benchmark's original expected scope.

  3. Long-horizon coverage is somehow limited. Even though WBench is designed as multi-turn benchmark, the dataset only has 3.7 turns in average, and only 12% of cases contain 5-9 turns. The longer 20-turn analysis is restricted to two navigation scenarios. Given that one of the paper’s main findings is huge degradation over turns, I think the current benchmark's multi-turn may still be too short to robustly characterize and analysis long-horizon memory and consistency.

Questions for authors
  1. How do the authors plan to deal with the problem when hosting the leaderboard in long-term as some of the metrics are depended on VLM that might have API changes? Several metrics rely on API-based VLM, and the authors note that hosted model or API changes may alter scores. It is more of a common issue for VLM-based evaluation. So I want to discuss with the authors about how to address this problem?

  2. Navigation magnitude. Could the authors quantify or give some intuitions on how rankings change if navigation magnitude is also evaluated?

2 comments

Thanks for the clear response, the reviewer address my concerns, and I would like to keep my original scores.

Thank you for your feedback and constructive suggestions. We appreciate your time and consideration.

Reviewer_ghANOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WBench, a benchmark for evaluating interactive world models across video quality, setting adherence, interaction adherence, consistency, and physics compliance. It contains 289 cases with 1,058 interaction turns covering navigation, subject actions, event editing, and perspective switching. A unified navigation protocol supports text, camera-pose, and discrete-action inputs. The authors evaluate 20 models using 22 automatic metrics, analyze capability trade-offs and degradation across turns, and compare automated model rankings with human preferences.

Critical strengths
  1. The evaluation suite is comprehensive and detailed, with 22 metrics covering five complementary dimensions. This allows WBench to distinguish specific failures in control, consistency, and physics that video quality metrics alone may overlook.
  2. The comparison with human judgments provides empirical support for the automated evaluation. The strong agreement at the model-ranking level increases confidence that the benchmark captures differences meaningful to human observers.
Critical weaknesses

None

Questions for authors
  1. Perspective consistency measures how stable the subject's position and visibility are across frames. However, the subject's position may change naturally when the camera moves, such as when the camera orbits around a person. How does the metric distinguish such valid changes from unwanted subject drift?
  2. Navigation Score adjusts the reference trajectory based on the model's predicted motion. While this helps compare models with different control interfaces, it may not fully capture errors in movement distance or rotation angle. How do the authors validate the reliability of this metric when comparing navigation performance across different models?
Reviewer_kSSyOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Wbench is a benchmark for interactive video would model with 289 cases and more than 1000 turns that covers navigation, action, event editing, perspective switching. The evaluation utilized expert vision models, VLM based judges, and a finetuned Qwen3 30B scorer. The authors found that no single model outperformed the others across every dimension, and that navigation performance declined most significantly in relation to the number of turns required.

Critical strengths
  1. It covers a total of 20 models, from text driven and camera controlled systems to those triggered by specific actions
  2. Reliability has been well documented. Each case was generated and scored twice, and a confidence intervals are provided.
  3. Very heavy and comprehensive evaluation, enough contribution.
Critical weaknesses
  1. it is better if you could include the length of video comparison in table 1.
  2. The evaluation of multi-turn interactions covers only a subset of capabilities. Specifically, features such as agent actions, event editing, and viewpoint switching were tested on only 9 text-driven models, while the remaining 11 models were evaluated solely on their navigation abilities.
  3. Mostly, VLM scoring is using doubao-seed-2-0-lite, another VLM from anther model family would help to reduce the dependency of a single VLM preference.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.