- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PROWBench to check whether the generated video from a video generator/world model aligns with the program (game engine). They synthesize 170 episodes in a game engine, then feed those inputs to 11 video generator/world models and calculate a bunch of metrics against the engine states, like position checking by SAM3, camera motion by ViPE, and two new VLM-as-judge metrics to check events (ISR & LRA).
Critical strengths
-
Using the game engine's state as gt for benchmarking makes sense since it provides perfect ground truth (even though there might still be problems with the metric). I think this is the right way of thinking about evaluating world models/video generation models.
-
I think the failure analysis is comprehensive and thoughtful. The paper provides failure mode analysis which can be transferred when analysing other models. Like outfit swapping between people or the wrong direction of the entity.
Critical weaknesses
-
I think the most critical weakness is that the two novel metrics proposed by the authors also use a VLM as a judge, and we also haven't tested whether the metrics align with human preferences. So we don't know if the metric is better than the CLIP-based metric the authors criticized in the first place in the paper.
-
MiniMax-H3 plays several roles in the paper: (1) generates all first frames; (2) was used to curate that 90-case set; (3) was used to pick the proxy type. I'm not so sure if they did any bias analysis for using the MiniMax model in all those applications, since there could be some confounding factors about this model. And MiniMax is also a model that's being evaluated in this benchmark.
Questions for authors
-
I think the most important thing the authors should address is demonstrating whether the two metrics they proposed align with human judgment, so people can trust that the metrics make sense for evaluating world models or video generation models.
-
Also, I think they should answer the question of using MiniMax in many different ways, and whether the benchmark can fairly evaluate different models and disentangle the use of MiniMax while benchmarking MiniMax.