- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WORLDCAST contains 320 cases across eight task categories to evaluate multistep visual state generation from heterogeneous references. The paper compares 12 systems and analyzes language enhancement, interleaved generation, and tool-assisted iteration. Human comparisons, multiple judges, and repeated generations validate the evaluation.
Critical strengths
- Meaningful tasks. The benchmark tests reference binding, action ordering, and object consistency across multiple steps, exposing failures that static image evaluation can miss.
- Multiple validation checks. Human comparisons, different judges, and repeated generations support the stability of the main performance gaps.
Critical weaknesses
- Unclear contribution. WorldReasonBench [1] already evaluates world-state prediction, while UEval [2] covers procedural multimodal generation. The paper does not clearly establish the additional diagnostic value of combining heterogeneous references with multistep generation. L3/L4 also lack testable classification criteria.
- Confounded experiments. Table 2 does not clearly control resolution per state or generation budget. The drop of 22.7 points therefore cannot be attributed directly to interleaved reasoning. Declining accuracy at later steps also leaves step difficulty and error propagation unresolved.
- Insufficient calibration against negative controls. A sequence can score 20 even with zero task completion and causal plausibility, provided visual consistency receives full credit. Without repeated-frame or shuffled-sequence baselines, the advantage over degenerate strategies and sensitivity to stalled or incorrectly ordered processes remain unclear.
Reference
[1] WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors.
[2] UEval: A Benchmark for Unified Multimodal Generation.
Questions for authors
- Which failure modes does WORLDCAST reveal that existing benchmarks cannot readily detect? What criteria distinguish L3 from L4?
- When only the referenced physical property or rule changes, while the scene and instruction remain fixed, does the output change correctly? Are paired results available?
- Does the performance gap in Table 2 persist after controlling resolution per state and reasoning and generation budgets?
- After supplying correct historical states to remove error propagation, does accuracy still decline as context length increases?
- Where applicable, what scores do repeated initial frames or repeated target references receive on each metric and overall?
- For human-verified correct sequences, do task completion and causal plausibility scores consistently decrease when steps are shuffled or critical stages are removed?
9 comments
Response to Reviewer otsS (2/4): L3/L4 Criterion and Paired Reference Variants
W1 & Q1 (cont.). What distinguishes L3 from L4?
We agree that the L3/L4 criterion deserves a more explicit statement. As Figure 3 shows, L3 and L4 differ in task setting, specifically in the role that the references play. We formalize the difference as follows:
| L3 | L4 | |
|---|---|---|
| Criterion: can the reference directly serve as the natural starting state? | Yes | No |
| Role of the reference | The starting world state, to be observed and rolled out | Carries information beyond the starting state (annotations, rules, target states, physical properties) |
| What the model must do first | Observe the initial state | Reason over the information-bearing references |
As noted in Section 2, L3/L4 classifies the complexity of task settings rather than ranking difficulty. We will state this criterion explicitly in the revision.
Q2. Do outputs change correctly when only the referenced property or rule changes?
We thank the reviewer for this valuable question, which we also took into account when designing WorldCast. If a model actually uses a reference, its output should change accordingly when the referenced property or rule changes; if it does not, the model has ignored the reference, and this should be penalized by our judge. In designing the cases, we intended that a natural rollout without relying on the references would generally not lead to the intended outcome, and the judge-only hints specify the states that the references determine. Outputs that neglect a reference therefore fail to match these states and lose credit.
In most cases the reference is an integral part of the task itself, so replacing it directly would make the task implausible or require redesigning the entire case. As the reviewer's question concerns physical properties or rules, we focused on cases in this domain and selected 5 of them in which the reference can be isolated relatively cleanly, constructing 10 paired variants in total. In 9 of the 10 pairs, the outputs change noticeably, and 8 of these changes are as expected; in the remaining two pairs, the output either does not reflect the change or reflects it incorrectly, which exposes a limitation of the model's own capability. We have updated the supplementary material with visual comparisons for all 10 pairs (ref_compare.pdf). If the reviewer finds this analysis informative, we will add it to the appendix.
Response to Reviewer otsS (3/4): Budget Control and Error Propagation
Q3 & W2 (part 1). Does the Table 2 gap persist after controlling resolution and budget?
We did control output resolution; the detailed settings are as follows, and we will document them in the appendix. For most four-state outputs, we use a 2× aspect ratio so that the output resolution per state can be matched. For certain input resolutions and for 2- or 6-state outputs, closed-source APIs only support specific resolutions, so we selected the closest available setting. As a benchmark, we control the final outputs and the way each model is invoked, whereas the internal reasoning and generation budgets of closed-source systems cannot be fully controlled.
Regarding the interpretation of the 22.7-point gap, our claim in Section 4.3.2 is that it reflects a difference in how the model is invoked, i.e., interleaved versus grid generation, rather than an effect attributable solely to interleaved reasoning. To address such concern directly, we further ran a controlled study on 198 cases in which the output budget is exactly matched between the two formats: the grid is generated at 2× resolution and then split into individual states, so each state has the same final resolution as the corresponding interleaved output. The gap narrows from 22.7 to 19.7 points but remains large:
| GPT-Image-2.5 | Interleaved | Grid | Gain of interleaved |
|---|---|---|---|
| Overall | 74.3 | 54.6 | +19.7 |
Q4 & W2 (part 2). Does accuracy still decline when correct historical states are supplied?
We agree that separating step difficulty from error propagation could be of diagnostic value. Following the reviewer's suggestion, we ran this analysis on the 41 human-validated correct sequences (generated by GPT-Image-2/2.5, also requested from Q6), supplying Nano Banana Pro with those correct historical states. The step accuracy (%) is shown below:
| Nano Banana Pro | Step 1 | Step 2 | Step 3 | Step 4 |
|---|---|---|---|---|
| Original | 75.6 | 65.9 | 56.1 | 58.5 |
| With correct history | 80.5 | 78.0 | 73.2 | 73.2 |
Accuracy improves but still declines across steps (80.5 → 73.2), suggesting that both error propagation and step difficulty contribute to the decline; the latter may arise from the increasing number of historical images at later steps, which adds to the information the model must process. However, we did not adopt this as the main protocol, for two reasons:
- Correct history cannot fully reproduce a model's own rollout. Open-ended tasks usually have no unique "correct" history. Even when correct historical images are supplied as suggested, the reasoning that produced them cannot be extracted, so this setting is not true teacher forcing. Moreover, for this controlled setting, each call requests the next state rather than all four states at once as in the original rollout, which may affect the model's internal generation pattern and partly explain the gap even at Step 1 (80.5 vs. 75.6).
- Our protocol is already aligned with what WorldCast aims to evaluate: coupled multimodal understanding and generation in realistic usage, where a model must build on its own previous outputs and reasoning. Evaluating the outcome of each model's own rollout directly reflects how understanding, reasoning, and generation jointly unfold across steps. This is also in line with related multi-step evaluation settings such as world-model benchmarks; for example, WBench [1] rolls out multi-turn interactions from each model's own previous outputs without supplying ground-truth intermediate states.
References
[1] Kaining Ying et al. WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation. arXiv:2605.25874, 2026.
Response to Reviewer otsS (1/4): Contribution of WorldCast
We thank the reviewer for the detailed and constructive comments, and for recognizing the meaningfulness of our tasks and validation checks. Our responses are below, and we kindly ask the reviewer to consider raising the score if they find them helpful.
W1 & Q1. What is the contribution of WorldCast beyond existing benchmarks?
The core contribution of WorldCast is its focus on dynamically entangled multimodal understanding and generation. UEval [1] targets unified multimodal generation, with a focus on knowledge and verbal reasoning. WorldReasonBench [2] evaluates video generators that start from an initial world state (TI2V). WorldCast studies how visual understanding, textual reasoning/planning, and generation are dynamically coupled across multiple generation steps, which neither benchmark addresses. Heterogeneous references are provided precisely to probe this coupled capability.
This problem remains under-studied, as prior work mainly examines whether a model possesses certain knowledge, or whether it can faithfully reproduce pixel-level content (appearance, identity, style) from references. In WorldCast, references and task types are designed so that a reference can carry rules, annotations, target states, or physical properties that should take effect at different time steps. This tests whether a model can integrate several capabilities: understanding the reference, planning when and how to apply it, and generating accordingly, sometimes even with agentic behavior. WorldCast thus provides a more holistic evaluation of multimodal intelligence.
Under this focus, WorldCast exposes distinct diagnostic patterns, of which we highlight two representative ones. First, frontier models still often fail to utilize complex references. In Figure 4 (right), the target reference specifies the structure of the windmill, and the task requires attaching the blades in a specific order: even GPT-Image-2.5 drifts from the referenced shape at the first step, and Wan 3.0 attaches the blades in an incorrect order. Many models also fail to reflect complex physical properties specified by references, such as adhesion (Figure 14). Second, different combinations of components behave differently under the stress of heterogeneous references and multi-step generation. Such challenging settings require tightly coupled capabilities, and in particular strong language-side reasoning to orchestrate them; WorldCast diagnoses how these components interact with the generator through language-model enhancers (Section 4.3.1), interleaved generation (Section 4.3.2), and agentic harnesses (Section 4.3.3, Table 3).
References
[1] Bo Li et al. UEval: A Benchmark for Unified Multimodal Generation. arXiv:2601.22155, 2026.
[2] Keming Wu et al. WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors. arXiv:2605.10434, 2026.