acceptodds
Withdrawn from ICLR 2027

PROWBench: An Extensible Benchmark for Programmable World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However, their visual adherence to explicit rules and interactions remains insufficiently evaluated. Existing benchmarks emphasize video quality and camera controllability, with limited support for evaluation grounded in replayable world records. We introduce PROWBench, comprising 170 programmatically constructed episodes and 600 proxy videos covering diverse scenes and interactions. An extensible framework constructs scenes, controls behaviors, and renders three synchronized representations per camera view, including coarse 3D, semantic proxy, and colored oriented bounding boxes, from shared state and interaction records. The benchmark covers first- and third-person perspectives, with synchronized multi-view observations available for a subset of episodes. Two VLM-based metrics, Logic–Render Alignment and Interaction Success Rate, assess prompt-rule adherence and the visual realization of timestamped engine-recorded events, respectively.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Generative Rendering, Controllable Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g5p3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PROWBench to check whether the generated video from a video generator/world model aligns with the program (game engine). They synthesize 170 episodes in a game engine, then feed those inputs to 11 video generator/world models and calculate a bunch of metrics against the engine states, like position checking by SAM3, camera motion by ViPE, and two new VLM-as-judge metrics to check events (ISR & LRA).

Critical strengths
  1. Using the game engine's state as gt for benchmarking makes sense since it provides perfect ground truth (even though there might still be problems with the metric). I think this is the right way of thinking about evaluating world models/video generation models.

  2. I think the failure analysis is comprehensive and thoughtful. The paper provides failure mode analysis which can be transferred when analysing other models. Like outfit swapping between people or the wrong direction of the entity.

Critical weaknesses
  1. I think the most critical weakness is that the two novel metrics proposed by the authors also use a VLM as a judge, and we also haven't tested whether the metrics align with human preferences. So we don't know if the metric is better than the CLIP-based metric the authors criticized in the first place in the paper.

  2. MiniMax-H3 plays several roles in the paper: (1) generates all first frames; (2) was used to curate that 90-case set; (3) was used to pick the proxy type. I'm not so sure if they did any bias analysis for using the MiniMax model in all those applications, since there could be some confounding factors about this model. And MiniMax is also a model that's being evaluated in this benchmark.

Questions for authors
  1. I think the most important thing the authors should address is demonstrating whether the two metrics they proposed align with human judgment, so people can trust that the metrics make sense for evaluating world models or video generation models.

  2. Also, I think they should answer the question of using MiniMax in many different ways, and whether the benchmark can fairly evaluate different models and disentangle the use of MiniMax while benchmarking MiniMax.

Reviewer_FyfQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PROWBench evaluates whether video models faithfully depict a world whose states and interactions are specified by a program. Its 170 cases provide replayable state and event records, paired proxy representations, and synchronized views for a subset of episodes. The benchmark combines geometric measures with VLM judgments of whether prescribed actions and outcomes appear at the expected times. Experiments examine different control inputs and appearance references, showing that visually plausible outputs can still contain errors in entity identity, interaction outcomes, and consistency across time and viewpoints.

Critical strengths
  1. PROWBench allows different views and proxy representations to be evaluated against the same recorded world evolution. Replayable states, entity identities, and timestamped events provide concrete references for checking prescribed motion and interaction outcomes. This creates a useful setting for controlled comparisons and failure diagnosis in programmable-world rendering.

  2. The qualitative examples and separate analyses of geometry, appearance, and events make distinct failure modes visible. Cases where characters occupy the expected positions but receive incorrect appearances or orientations help identify concrete targets for improving entity binding and consistency.

Critical weaknesses
  1. The new event metrics lack sufficient validation against human judgments. Engine records specify what should happen, but do not establish whether the VLM reliably recognizes the actions and outcomes in generated videos. Appearance scores also exclude unjudgeable observations without reporting the corresponding coverage in Table 5. These gaps limit confidence in the automatic scores and model differences, although the geometric measurements and qualitative observations remain useful.

  2. Several comparisons mix model capability with input and selection effects. Models receive different control information, the two first-frame settings use disjoint scenes, and MiniMax-H3 contributes to both evaluation-set curation and initialization. The results characterize the evaluated pipelines, but provide limited evidence for attributing gains to particular representations or conditioning choices.

Questions for authors

How well do the event scores agree with human annotations, and do those annotations support the main differences between models? Please also report judgeable coverage for the appearance scores.

How were MiniMax-H3 outputs used to curate the evaluation set? Were cases selected based on its output quality, and do the main findings hold on a subset selected independently of those outputs?

Reviewer_cXRSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PROWBench asks whether video models faithfully render the explicit states, rules, and timestamped interactions of programmable world models, rather than merely producing plausible video. It provides 170 engine-recorded episodes rendered as 600 synchronized proxy videos across three representations and two perspectives, plus two VLM-based metrics (Logic–Render Alignment, Interaction Success Rate) grounded in replayable world records. Across eleven models, proxy-conditioned models track entities far better than camera-conditioned ones, but appearance binding, facing direction, long-horizon state preservation, and cross-view identity remain unreliable.

Critical strengths
  1. The evaluation question is well defined and genuinely underserved. Faithful visual realization of engine-recorded states, rules, and interactions is exactly what programmable world models need measured. Table 3 shows no prior benchmark combines state records, replay, multi-view, and paired representations.

  2. Three synchronized representations from one shared state, synchronized multi-view renders of one world, and first-frame versus no-first-frame splits each isolate a distinct factor. This design produces real findings, not just scores: geometric multi-view agreement without identity agreement (Table 5), and long-horizon reappearance failure despite short-clip control (Table 4).

  3. Judge limitations, pose-estimator imperfection, source-defined labels without physical causality, and concrete failure modes (swapped outfits, wrong facing direction) are stated openly with qualitative evidence, and the domain-specific case study (Table 7) shows the event-level metrics separating rough placement from actual event execution.

Critical weaknesses
  1. The two signature metrics are VLM judgments that the authors themselves report as uncalibrated. ISR and LRA carry the paper's central claim, yet both delegate to Qwen3.6-27B with greedy decoding, and the paper states the judgments "have not been systematically calibrated against human annotations." There is no human-agreement study, no second judge, and no error analysis of when the judge fails; the only validity evidence is the single qualitative case in Figure 1. For a benchmark, the metrics are the product, and unvalidated metrics leave every downstream ranking provisional.

  2. The dataset is small where the interesting claims live, and nothing quantifies ranking stability. Long-horizon rests on 10 recordings and multi-view on 10 scenes; 570 of 600 videos are five seconds. Each method produces one output per case, and tables report point estimates without seeds, variance, or confidence intervals. The authors themselves call the long-horizon numbers "tendencies rather than stable rankings," and state persistence rests on 17–21 judge-scored events per method. Gaps such as 0.494 versus 0.518 BBox IoU in Table 1 are within what single-sample noise could plausibly produce.

Questions for authors
  1. Can you validate ISR and LRA against human annotations on a sampled subset (say 150–200 events and segments), report judge–human and inter-judge agreement across at least two VLMs, and show sensitivity to judge prompting?

  2. How stable are the rankings? Please report confidence intervals or seed replicates for Tables 1–2, and either enlarge the long-horizon and multi-view subsets or state explicit power bounds for the claims drawn from them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.