- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce WBench, a benchmark for evaluating interactive world models across five complementary dimensions through explicit world settings and multi-turn interactions. In order to do this, the authors use careful design and specifications, and collect data that spans diverse scenarios from multiple levels and perspectives, and I think it basically cover most if not all of the world generation settings. Experiments on 20 representative world models reveal that no model excels in all dimensions. Thus the authors use fine-grained profiles to help identify which capabilities fail under a given interaction, rather than treating visual quality or a single aggregate score as a substitute for world-model competence. This makes sense to me because I do think reporting dimension-level results alongside overall comparisons gives us a more detailed picture as for which model or architecture design works for a certain evaluation dimension or a certain type of interaction.
Critical strengths
-
A new direction of evaluating interactive world-model. The benchmark goes beyond evaluating single turn modality. The authors propose a benchmark for multi-turn world generation evaluation. The authors do carefully design to support the evaluation through four interaction categories, with both first- and third-person perspectives and multi-turn design. I love the idea the authors proposed that the benchmark score is merged into a single overall one, and different dimension of analysis offer a more comprehensive evaluation on the model.
-
Multi-dimension evaluation suite with thorough empirical study and metric validation. The benchmark proposes 5 dimensions: video quality, setting, interaction, consistency, physical, with 22 metrics in total. The evaluation covers 20 recent models. The authors also make a substantial effort to validate the benchmark metrics, with 400 annotators and 13,515 pairwise comparisons used for human preference evaluation. These analyses make the benchmark more convincing.
Critical weaknesses
-
Limited discriminative power of many metrics. I agree with the authors' claim that "No model or paradigm dominates uniformly". However, I notice that several metrics appear highly concentrated or close to saturation across models, which might limits their discrimination ability. For example, Aesthetic score, imaging, flickering, smoothness metrics in video quality dimension, the scores are so close that we can't really safely define which model is better than the other. Also for background, gated spatial, subject, geometric, photometric metrics in consistency dimension, and for visual plausibility metric in physical dimension.
-
Imbalanced coverage of interaction types. 57% of the dataset is navigation, with only 6% for perspective switching. I wonder if it raises concerns about whether the reported rankings are representative for the benchmark's original expected scope.
-
Long-horizon coverage is somehow limited. Even though WBench is designed as multi-turn benchmark, the dataset only has 3.7 turns in average, and only 12% of cases contain 5-9 turns. The longer 20-turn analysis is restricted to two navigation scenarios. Given that one of the paper’s main findings is huge degradation over turns, I think the current benchmark's multi-turn may still be too short to robustly characterize and analysis long-horizon memory and consistency.
Questions for authors
-
How do the authors plan to deal with the problem when hosting the leaderboard in long-term as some of the metrics are depended on VLM that might have API changes? Several metrics rely on API-based VLM, and the authors note that hosted model or API changes may alter scores. It is more of a common issue for VLM-based evaluation. So I want to discuss with the authors about how to address this problem?
-
Navigation magnitude. Could the authors quantify or give some intuitions on how rankings change if navigation magnitude is also evaluated?
2 comments
Thank you for your feedback and constructive suggestions. We appreciate your time and consideration.
Thanks for the clear response, the reviewer address my concerns, and I would like to keep my original scores.