- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoWorld Reasoner, an agentic framework that exposes a video world model (VWM) to a VLM through tools for four visual operations (view change, interpolation, continuation, intervention), together with skills and a verifier. A "co-evolution" procedure uses a GPT-5.6 supervisor to revise the tools, skills, and execution code over four reasoning stages, based on 120 development examples; both models stay frozen. On seven benchmarks, the method improves over direct GPT-5.6 reasoning by 1.40 points on average, with the largest gains on MMSI (+4.54) and ERQA (+6.75).
Critical strengths
- The motivation is clear: different reasoning problems need different kinds of visual imagination, so a VWM should be called selectively rather than through a fixed novel-view pipeline. Tool-usage statistics (Fig. 3b) show that the agent does select operations by task: view change and interpolation on MMSI, intervention and continuation on CausalSpatial, and rarely any tool on PerceptionBench.
- The comparison is reasonably controlled: five evolving baselines run on the same GPT-5.6 backbone with the same inputs, and the paper includes a no-VWM ablation, a VWM swap, and a transfer to Grok-4.5.
Critical weaknesses
- The evidence that VWM imagination drives the gains is weak. The average gain is small (+1.40), and there is no variance estimate or significance test; CausalSpatial (+0.07), PerceptionBench (+0.51), and SL-MCQ (+0.71) are within noise. On CausalSpatial, the benchmark that most directly targets intervention and continuation, Occlusion drops by 4.76 points, so the "tool gains" in Fig. 3a mostly recover losses introduced by the no-VWM harness. On ERQA, CoWorld (58.50) trails three baselines that use no VWM (Meta-Harness 60.00, GEPA 59.75, ReasoningBank 59.00), which suggests that the headline ERQA gain comes from the multi-stage agent rather than from imagination.
- The comparisons are not compute-matched and the baselines are weak. CoWorld uses up to 22 model turns plus video generation and verifier calls per item, whereas the direct baseline uses a single call; latency, cost, and call counts are not reported. All five evolving baselines score below direct GPT-5.6, which suggests overfitting to the 120 development examples or weak adaptation. In addition, the same GPT-5.6 serves as agent, supervisor, and judge. Novelty is mainly a combination of VWM tool use (MindJourney, AVIC) and harness evolution (Meta-Harness, GEPA), and "VLM-VWM co-evolution" overstates what changes, since only the harness evolves.
Questions for authors
- Can you report mean ± std over multiple runs or seeds, along with significance tests, for the main table?
- What are the average model calls, VWM calls, latency, and cost per item, and how does the method compare with a compute-matched baseline (e.g., GPT-5.6 self-consistency)?
- Why does Occlusion on CausalSpatial drop below direct reasoning when VWM tools are used? How often are the generated videos judged faithful to the request?
- For Grok-4.5, how does the transferred harness compare with evolving from scratch on Grok and with the evolving baselines on Grok?
9 comments
Response to Reviewer CsEs (2/6)
W1: “The evidence that VWM imagination drives the gains is weak. ... On ERQA, CoWorld (58.50) trails three baselines that use no VWM (Meta-Harness 60.00, GEPA 59.75, ReasoningBank 59.00), which suggests that the headline ERQA gain comes from the multi-stage agent rather than from imagination.”
Response: Thank you for raising the question of whether VWM imagination drives the gains. To separate its role from what the multi-stage agent provides, we first examine it at test time and during co-evolution with an additional controlled comparison, and then discuss the ERQA result.
We evolved the same initial harness with and without VWM access, using the same 120 development examples, rounds and revision budget, and evaluated each harness on all 11,544 held-out items with the VWM disabled at test time (w/o VWM); the harness co-evolved with the VWM was also evaluated with the VWM enabled. Besides overall accuracy, we report the 249 MMSI positional-relationship questions involving the camera (Cam.–Cam., Cam.–Obj. and Cam.–Reg.; MMSI-Cam.) and the two SpatiaLab formats, which provided no development examples. Each entry is the accuracy on the full benchmark or subset:
| Harness (VWM at test time) | Overall accuracy (%) | MMSI-Cam. (%) | SL-MCQ (%) | SL-Open (%) |
|---|---|---|---|---|
| Initial harness (off) | 58.82 | 35.74 | 56.93 | 34.57 |
| Co-evolved without VWM (off) | 58.62 | 35.34 | 54.64 | 34.36 |
| Co-evolved with VWM (off) | 59.22 | 42.57 | 56.86 | 36.50 |
| Co-evolved with VWM (on) | 59.69 | 47.79 | 57.36 | 37.64 |
As shown in the table, (1) generated videos improve accuracy at test time. For the harness co-evolved with the VWM, enabling the VWM improves accuracy in every column: overall from 59.22% to 59.69%, on MMSI-Cam. from 42.57% to 47.79%, and also on both SpatiaLab formats. (2) Without generating videos, the harness still retains the visual reasoning capability acquired through co-evolution with the VWM. With the VWM disabled, the harness co-evolved with the VWM outperforms the harness co-evolved without it in every column. It also outperforms the initial harness overall, on MMSI-Cam. and on SL-Open. In contrast, the harness co-evolved without the VWM does not improve on the initial harness in any column.
ERQA and the evolving baselines. On ERQA, CoWorld's gain over direct reasoning replicates across three runs at +4.00 pp (Q1). The three cited baselines also perform well on ERQA, within 2–6 correct answers of CoWorld on its 400 items. Across the full evaluation, however, CoWorld outperforms all five evolving baselines on five benchmarks and reaches the highest overall accuracy (Table 1).
Together, these results highlight the role of VWM imagination in the gains. Beyond what the multi-stage agent provides, generated videos improve accuracy at test time, and co-evolution with the VWM gives the harness a visual reasoning capability that it retains even without generating videos. On ERQA, where CoWorld's gain replicates, the cited baselines are close, while across benchmarks CoWorld is stronger.
Q2: “The comparisons are not compute-matched ... CoWorld uses up to 22 model turns plus video generation and verifier calls per item, whereas the direct baseline uses a single call ... What are the average model calls, VWM calls, latency, and cost per item, and how does the method compare with a compute-matched baseline (e.g., GPT-5.6 self-consistency)?”
Response: Thank you for suggesting self-consistency as a compute-matched baseline. We report the average model calls, VWM calls, latency and cost per item, and compare CoWorld with GPT-5.6 self-consistency in two computation-matched settings: first at matched model calls, and then at matched cost.
Response to Reviewer CsEs (3/6)
Q2, continued: Comparison at matched model calls. We first report the measured usage: across all 11,544 held-out items, CoWorld averages 4.58 model calls per item, far below the 22-turn limit, and calls the VWM on 28.6% of them. We compare CoWorld with Direct reasoning@k (self-consistency), a majority vote over k GPT-5.6 samples at the agent's medium reasoning effort, on the 2,765 MMSI, ERQA and CausalSpatial items. On these items, CoWorld averages 5.21 model calls per item, between the budgets @5 and @9. The table reports accuracy on these items, with pooled accuracy and MMSI-Cam. results in the last two rows:
| Benchmark | Ours (5.21 calls/item) | @1 | @5 | @9 |
|---|---|---|---|---|
| MMSI | 42.89 | 40.10 | 40.10 | 40.21 |
| ERQA | 58.50 | 56.00 | 56.75 | 57.00 |
| CausalSpatial | 60.79 | 61.15 | 62.37 | 62.29 |
| Pooled | 54.18 | 53.02 | 53.74 | 53.78 |
| MMSI-Cam. | 47.79 | 35.74 | 36.95 | 38.15 |
As shown in the table, (1) additional samples change direct-reasoning accuracy little. On MMSI, direct-reasoning accuracy stays at 40.10–40.21% from @1 to @9. (2) CoWorld outperforms self-consistency at matched calls. Pooled over the three benchmarks, CoWorld outperforms both @5 and @9, and the gap is largest on MMSI-Cam., where CoWorld reaches 47.79% compared with 38.15% for @9. On CausalSpatial, self-consistency is slightly higher. Additional samples reason over the same input images, whereas CoWorld can request the visual evidence a question needs.
Comparison at matched cost. We further extend the comparison to 15 samples on the same items and measure cost per item at the rates we paid (USD 0.16 per million GPT-5.6 input tokens, USD 0.96 per million output tokens, and USD 0.0875 per video), including the agent, verifier and video generation. We take mean end-to-end latency from the main evaluation runs; for k > 1, latency is not applicable (–), as it depends on how many samples run concurrently. Accuracy here is the equal-weighted average over the three benchmarks (macro accuracy), whereas the pooled row above weights items equally:
| Method | Macro accuracy (%) | Model calls/item | VWM calls/item | Mean latency | Cost/item |
|---|---|---|---|---|---|
| Direct reasoning@1 | 52.42 | 1 | 0 | 61 s | $0.0008 |
| Direct reasoning@3 | 52.68 | 3 | 0 | – | $0.0024 |
| Direct reasoning@5 | 53.07 | 5 | 0 | – | $0.0040 |
| Direct reasoning@7 | 52.73 | 7 | 0 | – | $0.0056 |
| Direct reasoning@9 | 53.17 | 9 | 0 | – | $0.0073 |
| Direct reasoning@11 | 52.84 | 11 | 0 | – | $0.0091 |
| Direct reasoning@13 | 52.78 | 13 | 0 | – | $0.0108 |
| Direct reasoning@15 | 52.97 | 15 | 0 | – | $0.0126 |
| Ours (w/o VWM) | 53.73 | 4.49 | 0 | 280 s | $0.0064 |
| Ours | 54.06 | 5.21 | 0.64 | 351 s | $0.0662 |
As shown in the table, (1) the accuracy of self-consistency plateaus as its sampling cost increases. From @1 to @15, macro accuracy rises by only 0.55 pp, with fluctuations between budgets. (2) CoWorld w/o VWM outperforms every sampling budget. At about the cost of @9, it reaches 53.73%, and enabling the VWM further improves macro accuracy to 54.06%, the highest in the table.
These measurements show that CoWorld uses far fewer model calls than its turn limit, and that it outperforms self-consistency in both computation-matched settings: the harness co-evolved with the VWM outperforms every sampling budget even without generating a video, and generated videos add further accuracy. We will add the usage and cost table and the self-consistency results for k = 1 to 15 to a new Appendix A.12 on inference cost and sampling baselines.
Response to Reviewer CsEs (1/6)
Dear Reviewer CsEs,
Thank you for the detailed review and for recognizing the motivation for selective VWM use. We address your questions with additional experiments and analyses below.
In this response, we use direct reasoning to denote the GPT-5.6 baseline in Table 1 unless stated otherwise, CoWorld to denote our method with the final co-evolved harness, and w/o VWM to denote the same harness with VWM-driven tools and the verifier disabled at test time, as in the paper's "Ours (w/o VWM)".
Response: Thank you for suggesting these tests. We report paired significance tests for every benchmark in Table 1, and mean ± SD over three runs of both direct reasoning and CoWorld on MMSI and ERQA.
Significance tests for Table 1. We test each CoWorld − direct reasoning difference on the same runs as Table 1, with paired sign-flip tests that group SAT's two option orders and SpatiaLab's two formats by original question:
The paired tests show that the overall, MMSI, ERQA and Spin gains are significant. All eight differences are positive.
Mean ± SD over three runs. To test whether the gains replicate, we ran both direct reasoning and CoWorld two more times on MMSI and ERQA within the discussion period, for three runs including the Table 1 run. The table reports accuracy for each run, the mean ± SD, and CoWorld's mean gain over direct reasoning with its p-value:
The results indicate that the MMSI and ERQA gains replicate across runs. Both mean gains are significant, and every CoWorld run exceeds every direct-reasoning run on both benchmarks.
These tests show that the overall gain is significant and that the largest gains, on MMSI and ERQA, hold across independent runs. We will add the paired tests and three-run results to a new Appendix A.10 on statistical tests and answer changes.