acceptodds
Under review as a conference paper at ICLR 2027

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

Abstract

As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyond what the input images show. Many multimodal reasoning tasks require reasoning about unobserved scenes, transitions, and relations. Video World Models (VWMs) provide a natural unified engine for visual imagination by generating task-relevant visual content as additional evidence. However, existing approaches typically invoke a VWM through fixed pipelines centered on a predefined form of visual imagination, such as novel-view synthesis. To support flexible visual imaginations for diverse reasoning tasks, we introduce CoWorld Reasoner, an agentic framework that uses a VWM as a unified visual reasoning engine. Its Video World Model-Driven Harness exposes multiple visual operations through tools and uses skills to guide the VLM in deciding whether to imagine, selecting an operation, constructing a generation request, and interpreting the resulting visual evidence. We further introduce VLM-VWM Harness Co-Evolution, which jointly revises tools, skills, and execution code based on execution feedback while keeping both model weights fixed. Across seven benchmarks spanning spatial intelligence, temporal and causal reasoning, and perceptual reasoning, CoWorld Reasoner outperforms direct reasoning with the same VLM. The strongest variant improves on all seven benchmarks, including relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and outperforms all five evolving baselines using the same backbone across five benchmarks. Code and data are included in the Supplementary Material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Multimodal Reasoning, World Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CsEsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoWorld Reasoner, an agentic framework that exposes a video world model (VWM) to a VLM through tools for four visual operations (view change, interpolation, continuation, intervention), together with skills and a verifier. A "co-evolution" procedure uses a GPT-5.6 supervisor to revise the tools, skills, and execution code over four reasoning stages, based on 120 development examples; both models stay frozen. On seven benchmarks, the method improves over direct GPT-5.6 reasoning by 1.40 points on average, with the largest gains on MMSI (+4.54) and ERQA (+6.75).

Critical strengths
  1. The motivation is clear: different reasoning problems need different kinds of visual imagination, so a VWM should be called selectively rather than through a fixed novel-view pipeline. Tool-usage statistics (Fig. 3b) show that the agent does select operations by task: view change and interpolation on MMSI, intervention and continuation on CausalSpatial, and rarely any tool on PerceptionBench.
  2. The comparison is reasonably controlled: five evolving baselines run on the same GPT-5.6 backbone with the same inputs, and the paper includes a no-VWM ablation, a VWM swap, and a transfer to Grok-4.5.
Critical weaknesses
  1. The evidence that VWM imagination drives the gains is weak. The average gain is small (+1.40), and there is no variance estimate or significance test; CausalSpatial (+0.07), PerceptionBench (+0.51), and SL-MCQ (+0.71) are within noise. On CausalSpatial, the benchmark that most directly targets intervention and continuation, Occlusion drops by 4.76 points, so the "tool gains" in Fig. 3a mostly recover losses introduced by the no-VWM harness. On ERQA, CoWorld (58.50) trails three baselines that use no VWM (Meta-Harness 60.00, GEPA 59.75, ReasoningBank 59.00), which suggests that the headline ERQA gain comes from the multi-stage agent rather than from imagination.
  2. The comparisons are not compute-matched and the baselines are weak. CoWorld uses up to 22 model turns plus video generation and verifier calls per item, whereas the direct baseline uses a single call; latency, cost, and call counts are not reported. All five evolving baselines score below direct GPT-5.6, which suggests overfitting to the 120 development examples or weak adaptation. In addition, the same GPT-5.6 serves as agent, supervisor, and judge. Novelty is mainly a combination of VWM tool use (MindJourney, AVIC) and harness evolution (Meta-Harness, GEPA), and "VLM-VWM co-evolution" overstates what changes, since only the harness evolves.
Questions for authors
  1. Can you report mean ± std over multiple runs or seeds, along with significance tests, for the main table?
  2. What are the average model calls, VWM calls, latency, and cost per item, and how does the method compare with a compute-matched baseline (e.g., GPT-5.6 self-consistency)?
  3. Why does Occlusion on CausalSpatial drop below direct reasoning when VWM tools are used? How often are the generated videos judged faithful to the request?
  4. For Grok-4.5, how does the transferred harness compare with evolving from scratch on Grok and with the evolving baselines on Grok?
9 comments
AuthorsOpenReview ↗

Response to Reviewer CsEs (1/6)

Dear Reviewer CsEs,

Thank you for the detailed review and for recognizing the motivation for selective VWM use. We address your questions with additional experiments and analyses below.

In this response, we use direct reasoning to denote the GPT-5.6 baseline in Table 1 unless stated otherwise, CoWorld to denote our method with the final co-evolved harness, and w/o VWM to denote the same harness with VWM-driven tools and the verifier disabled at test time, as in the paper's "Ours (w/o VWM)".


Q1: “The average gain is small (+1.40), and there is no variance estimate or significance test; CausalSpatial (+0.07), PerceptionBench (+0.51), and SL-MCQ (+0.71) are within noise. ... Can you report mean ± std over multiple runs or seeds, along with significance tests, for the main table?”

Response: Thank you for suggesting these tests. We report paired significance tests for every benchmark in Table 1, and mean ± SD over three runs of both direct reasoning and CoWorld on MMSI and ERQA.

Significance tests for Table 1. We test each CoWorld − direct reasoning difference on the same runs as Table 1, with paired sign-flip tests that group SAT's two option orders and SpatiaLab's two formats by original question:

BenchmarkDifference (pp)p
MMSI+4.540.010
ERQA+6.750.009
Spin+1.510.043
SL-Open+1.430.21
SAT+1.330.61
SL-MCQ+0.710.56
PerceptionBench+0.510.56
CausalSpatial+0.071.00
All+1.400.0005

The paired tests show that the overall, MMSI, ERQA and Spin gains are significant. All eight differences are positive.

Mean ± SD over three runs. To test whether the gains replicate, we ran both direct reasoning and CoWorld two more times on MMSI and ERQA within the discussion period, for three runs including the Table 1 run. The table reports accuracy for each run, the mean ± SD, and CoWorld's mean gain over direct reasoning with its p-value:

BenchmarkMethodRun 1Run 2Run 3Mean ± SDMean gain (pp), p
MMSIDirect reasoning38.3540.0038.6639.00 ± 0.88–
CoWorld42.8942.5842.1642.54 ± 0.36+3.54, 0.001
ERQADirect reasoning51.7556.5054.7554.33 ± 2.40–
CoWorld58.5058.2558.2558.33 ± 0.14+4.00, 0.012

The results indicate that the MMSI and ERQA gains replicate across runs. Both mean gains are significant, and every CoWorld run exceeds every direct-reasoning run on both benchmarks.

These tests show that the overall gain is significant and that the largest gains, on MMSI and ERQA, hold across independent runs. We will add the paired tests and three-run results to a new Appendix A.10 on statistical tests and answer changes.

AuthorsOpenReview ↗

Response to Reviewer CsEs (2/6)

W1: “The evidence that VWM imagination drives the gains is weak. ... On ERQA, CoWorld (58.50) trails three baselines that use no VWM (Meta-Harness 60.00, GEPA 59.75, ReasoningBank 59.00), which suggests that the headline ERQA gain comes from the multi-stage agent rather than from imagination.”

Response: Thank you for raising the question of whether VWM imagination drives the gains. To separate its role from what the multi-stage agent provides, we first examine it at test time and during co-evolution with an additional controlled comparison, and then discuss the ERQA result.

We evolved the same initial harness with and without VWM access, using the same 120 development examples, rounds and revision budget, and evaluated each harness on all 11,544 held-out items with the VWM disabled at test time (w/o VWM); the harness co-evolved with the VWM was also evaluated with the VWM enabled. Besides overall accuracy, we report the 249 MMSI positional-relationship questions involving the camera (Cam.–Cam., Cam.–Obj. and Cam.–Reg.; MMSI-Cam.) and the two SpatiaLab formats, which provided no development examples. Each entry is the accuracy on the full benchmark or subset:

Harness (VWM at test time)Overall accuracy (%)MMSI-Cam. (%)SL-MCQ (%)SL-Open (%)
Initial harness (off)58.8235.7456.9334.57
Co-evolved without VWM (off)58.6235.3454.6434.36
Co-evolved with VWM (off)59.2242.5756.8636.50
Co-evolved with VWM (on)59.6947.7957.3637.64

As shown in the table, (1) generated videos improve accuracy at test time. For the harness co-evolved with the VWM, enabling the VWM improves accuracy in every column: overall from 59.22% to 59.69%, on MMSI-Cam. from 42.57% to 47.79%, and also on both SpatiaLab formats. (2) Without generating videos, the harness still retains the visual reasoning capability acquired through co-evolution with the VWM. With the VWM disabled, the harness co-evolved with the VWM outperforms the harness co-evolved without it in every column. It also outperforms the initial harness overall, on MMSI-Cam. and on SL-Open. In contrast, the harness co-evolved without the VWM does not improve on the initial harness in any column.

ERQA and the evolving baselines. On ERQA, CoWorld's gain over direct reasoning replicates across three runs at +4.00 pp (Q1). The three cited baselines also perform well on ERQA, within 2–6 correct answers of CoWorld on its 400 items. Across the full evaluation, however, CoWorld outperforms all five evolving baselines on five benchmarks and reaches the highest overall accuracy (Table 1).

Together, these results highlight the role of VWM imagination in the gains. Beyond what the multi-stage agent provides, generated videos improve accuracy at test time, and co-evolution with the VWM gives the harness a visual reasoning capability that it retains even without generating videos. On ERQA, where CoWorld's gain replicates, the cited baselines are close, while across benchmarks CoWorld is stronger.


Q2: “The comparisons are not compute-matched ... CoWorld uses up to 22 model turns plus video generation and verifier calls per item, whereas the direct baseline uses a single call ... What are the average model calls, VWM calls, latency, and cost per item, and how does the method compare with a compute-matched baseline (e.g., GPT-5.6 self-consistency)?”

Response: Thank you for suggesting self-consistency as a compute-matched baseline. We report the average model calls, VWM calls, latency and cost per item, and compare CoWorld with GPT-5.6 self-consistency in two computation-matched settings: first at matched model calls, and then at matched cost.

AuthorsOpenReview ↗

Response to Reviewer CsEs (3/6)

Q2, continued: Comparison at matched model calls. We first report the measured usage: across all 11,544 held-out items, CoWorld averages 4.58 model calls per item, far below the 22-turn limit, and calls the VWM on 28.6% of them. We compare CoWorld with Direct reasoning@k (self-consistency), a majority vote over k GPT-5.6 samples at the agent's medium reasoning effort, on the 2,765 MMSI, ERQA and CausalSpatial items. On these items, CoWorld averages 5.21 model calls per item, between the budgets @5 and @9. The table reports accuracy on these items, with pooled accuracy and MMSI-Cam. results in the last two rows:

BenchmarkOurs (5.21 calls/item)@1@5@9
MMSI42.8940.1040.1040.21
ERQA58.5056.0056.7557.00
CausalSpatial60.7961.1562.3762.29
Pooled54.1853.0253.7453.78
MMSI-Cam.47.7935.7436.9538.15

As shown in the table, (1) additional samples change direct-reasoning accuracy little. On MMSI, direct-reasoning accuracy stays at 40.10–40.21% from @1 to @9. (2) CoWorld outperforms self-consistency at matched calls. Pooled over the three benchmarks, CoWorld outperforms both @5 and @9, and the gap is largest on MMSI-Cam., where CoWorld reaches 47.79% compared with 38.15% for @9. On CausalSpatial, self-consistency is slightly higher. Additional samples reason over the same input images, whereas CoWorld can request the visual evidence a question needs.

Comparison at matched cost. We further extend the comparison to 15 samples on the same items and measure cost per item at the rates we paid (USD 0.16 per million GPT-5.6 input tokens, USD 0.96 per million output tokens, and USD 0.0875 per video), including the agent, verifier and video generation. We take mean end-to-end latency from the main evaluation runs; for k > 1, latency is not applicable (–), as it depends on how many samples run concurrently. Accuracy here is the equal-weighted average over the three benchmarks (macro accuracy), whereas the pooled row above weights items equally:

MethodMacro accuracy (%)Model calls/itemVWM calls/itemMean latencyCost/item
Direct reasoning@152.421061 s$0.0008
Direct reasoning@352.6830–$0.0024
Direct reasoning@553.0750–$0.0040
Direct reasoning@752.7370–$0.0056
Direct reasoning@953.1790–$0.0073
Direct reasoning@1152.84110–$0.0091
Direct reasoning@1352.78130–$0.0108
Direct reasoning@1552.97150–$0.0126
Ours (w/o VWM)53.734.490280 s$0.0064
Ours54.065.210.64351 s$0.0662

As shown in the table, (1) the accuracy of self-consistency plateaus as its sampling cost increases. From @1 to @15, macro accuracy rises by only 0.55 pp, with fluctuations between budgets. (2) CoWorld w/o VWM outperforms every sampling budget. At about the cost of @9, it reaches 53.73%, and enabling the VWM further improves macro accuracy to 54.06%, the highest in the table.

These measurements show that CoWorld uses far fewer model calls than its turn limit, and that it outperforms self-consistency in both computation-matched settings: the harness co-evolved with the VWM outperforms every sampling budget even without generating a video, and generated videos add further accuracy. We will add the usage and cost table and the self-consistency results for k = 1 to 15 to a new Appendix A.12 on inference cost and sampling baselines.

Reviewer_wM28OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoWorld Reasoner couples a vision-language model (VLM) to a video world model (VWM) through an editable harness of tools, skills, a verifier, and execution code. The agent can request viewpoint changes, temporal interpolation, sequence continuation, or interventions, then interpret the generated frames alongside the observed images. A supervisor revises the harness in four stages using execution feedback while keeping both models' weights fixed. The harness is evolved on 120 development examples and frozen for evaluation. Experiments compare against direct reasoning and five evolving agents across seven benchmarks, with additional ablations and VLM/VWM variants. The strongest configuration reports item-weighted accuracy of 59.69%, versus 58.29% for direct GPT-5.6, with positive point estimates on all seven benchmarks.

Critical strengths
  1. The method makes the generation interface a concrete object of optimization. Sections 3.1-3.2 and Algorithm 1 connect operation selection, request construction, interpretation, and answer integration within a shared framework. Comparing stage candidates from the same incoming histories is a useful design choice, supported by candidate validation and harness snapshot code in the supplement. This offers a practical basis for studying how fixed generative models can support reasoning through changes to their interfaces.

  2. The reported gains provide a meaningful empirical motivation. Table 1 shows improvements of 4.54 percentage points on MMSI and 6.75 points on ERQA over the same VLM's direct reasoning. The evaluation spans seven benchmarks, and SAT, SpatiaLab, and ERQA contribute no development examples (Table 3). In particular, the ERQA result provides encouraging evidence that the final harness can help beyond the benchmarks used for adaptation.

Critical weaknesses
  1. The added computation lacks a convincing benefit-cost justification, and the contribution of co-evolution remains unclear. The direct baseline uses one stage, whereas CoWorld adds four-stage execution, evolved guidance, optional video generation and verification (Appendix A.3). Table 5 gives model-turn limits of 4/6/6/6 and a 4,096-token output cap per call, but actual per-item token usage, latency and inference cost are not reported. These limits are not measured consumption, and Figure 3b shows selective tool use. Nevertheless, outside MMSI and ERQA, the reported gains are only 0.07-1.51 percentage points, leaving the return on the additional computation unestablished. The five evolving baselines lack VWM access; the w/o VWM ablation disables both generation and verification within the final evolved harness and reports selected differences and recovery rates. No initial-versus-final harness comparison holds tools and inference budget fixed. Thus, neither the incremental value of co-evolution nor an advantage over simpler reasoning at comparable cost is established, even if the reported point estimates hold.

  2. Higher point estimates on every benchmark do not establish a reliable improvement across tasks. Tables 1 and 3 imply just one additional correct answer on CausalSpatial (847 to 848 of 1,395), four on SAT (265 to 269 of 300), and fifteen on PerceptionBench (1,510 to 1,525 of 2,970). If Figure 4 uses the same run, CausalSpatial's 28.1% recovery rate corresponds to correcting 154 baseline errors while losing 153 previously correct answers. The authors acknowledge similar overall accuracy in Section 4.3, but do not report paired uncertainty or variation across repeated runs. SAT's 300 items are two permutations of 150 questions, and SpatiaLab's two columns use the same 1,400 questions, so uncertainty estimates must preserve these dependencies. The all-seven improvement is descriptively correct; the available evidence does not establish that each gain is robust. This does not show that the gains are noise, and the larger MMSI and ERQA improvements remain encouraging.

Questions for authors
  1. What do available measurements of per-item inference cost and matched-budget comparisons with the initial harness and simpler reasoning baselines show about the incremental value of co-evolution?

  2. Can paired corrected/newly incorrect counts and uncertainty estimates from the same runs as Table 1, grouping repeated formats by original question, establish which benchmark gains are robust?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wM28 (1/3)

Dear Reviewer wM28,

Thank you for recognizing the generation interface as a concrete object of optimization and the gains beyond the development benchmarks.

In this response, we use direct reasoning to denote the GPT-5.6 direct reasoning baseline (its Table 1 run unless stated otherwise), and w/o VWM to denote CoWorld with all VWM-driven tools and the verifier disabled at test time, keeping the skills and four-stage execution, as in the paper's "Ours (w/o VWM)".


Q1: “The added computation lacks a convincing benefit-cost justification, and the contribution of co-evolution remains unclear. ... What do available measurements of per-item inference cost and matched-budget comparisons with the initial harness and simpler reasoning baselines show about the incremental value of co-evolution?”

Response: Thank you for this question. To assess the benefit relative to the added cost and the incremental value of co-evolution, we first report the measured per-item inference cost and compare CoWorld with majority voting at matched cost. We then compare the initial and the final harness under the same limits, and finally compare co-evolution with and without VWM access, which the five evolving baselines lack.

Per-item cost at matched budgets. We measured cost on the 2,765 MMSI, ERQA and CausalSpatial items for CoWorld with and without the VWM and for Direct reasoning@k, a majority vote over k GPT-5.6 samples at the same medium reasoning effort as the agent. Costs use the rates we paid (USD 0.16 per million GPT-5.6 input tokens, USD 0.96 per million output tokens, and USD 0.0875 per video) and include the agent, verifier and video generation. We take mean end-to-end latency from the main evaluation runs; for Direct reasoning@k with k > 1, latency is not applicable (–), as it depends on how many samples run concurrently. The table reports macro accuracy, model calls and tokens of the reasoning model (both excluding the verifier), latency and cost per item:

MethodMacro accuracy (%)Model calls/itemInput / output tokens per itemMean latencyCost/item
Direct reasoning@152.4211,959 / 51761 s$0.0008
Direct reasoning@352.6835,879 / 1,551–$0.0024
Direct reasoning@553.0759,797 / 2,563–$0.0040
Direct reasoning@752.73713,729 / 3,592–$0.0056
Direct reasoning@953.17918,056 / 4,622–$0.0073
Direct reasoning@1152.841122,790 / 5,656–$0.0091
Direct reasoning@1352.781327,506 / 6,682–$0.0108
Direct reasoning@1552.971532,238 / 7,718–$0.0126
Ours (w/o VWM)53.734.4922,287 / 2,936280 s$0.0064
Ours54.065.2138,874 / 2,833351 s$0.0662

As shown in the table, (1) the accuracy of direct reasoning plateaus as its sampling budget increases. From @1 to @15, macro accuracy rises by only 0.55 pp, with fluctuations between budgets. (2) CoWorld w/o VWM outperforms every sampling budget. Disabling the VWM cuts cost per item by about 90%, to about the cost of Direct reasoning@9, and the full system reaches the highest accuracy.

AuthorsOpenReview ↗

Response to Reviewer wM28 (2/3)

Q1, continued: Initial versus final harness. To measure what co-evolution adds under the same limits, we ran the initial harness of Algorithm 1 and the final harness on all 11,544 held-out items, with the same GPT-5.6 deployment, VWM, verifier and model-turn limits. The table reports per-benchmark accuracy and model calls per item; bold differences have paired 95% CIs that exclude zero:

BenchmarkInitial (%)Final (%)Difference (Final − Initial, pp)Model calls/item (Initial → Final)
MMSI37.9442.89+4.954.24 → 5.76
CausalSpatial57.6360.79+3.154.21 → 5.03
SL-Open34.6437.64+3.004.14 → 4.32
SAT87.6789.67+2.004.06 → 4.91
PerceptionBench50.0051.35+1.354.10 → 4.12
Spin82.9183.76+0.854.16 → 4.69
SL-MCQ56.7157.36+0.644.09 → 4.33
ERQA58.5058.50+0.004.12 → 4.51
All57.8659.69+1.844.14 → 4.58

The table shows that co-evolution improves the initial harness under the same limits. Overall accuracy rises from 57.86% to 59.69%, no benchmark declines, and the gains on MMSI, CausalSpatial and SL-Open are significant, as is the overall gain. Within the same limits, model calls per item rise from 4.14 to 4.58 on average.

Co-evolution with and without VWM access. The five evolving baselines revise the harness without VWM access. To test what VWM access adds, we ran an additional controlled comparison: we evolved the same initial harness with and without it, using the same 120 development examples, rounds and revision budget, and evaluated each harness on all 11,544 held-out items with the VWM disabled at test time (w/o VWM); the harness co-evolved with the VWM was also evaluated with the VWM enabled. Besides overall accuracy, we report the 249 MMSI positional-relationship questions involving the camera (Cam.–Cam., Cam.–Obj. and Cam.–Reg.; MMSI-Cam.), the two SpatiaLab formats, which provided no development examples, and CausalSpatial, a benchmark of reasoning under object interventions. Each entry is the accuracy on the full benchmark or subset:

Harness (VWM at test time)Overall accuracy (%)MMSI-Cam. (%)SL-MCQ (%)SL-Open (%)CausalSpatial (%)
Initial harness (off)58.8235.7456.9334.5759.71
Co-evolved without VWM (off)58.6235.3454.6434.3660.43
Co-evolved with VWM (off)59.2242.5756.8636.5058.49
Co-evolved with VWM (on)59.6947.7957.3637.6460.79

The results support two points. (1) Co-evolution with VWM access yields a stronger harness. With the VWM enabled, the harness co-evolved with the VWM outperforms the harness co-evolved without it in every column, from 58.62% to 59.69% overall. (2) This advantage holds even without generating videos. With the VWM disabled, the harness co-evolved with the VWM still outperforms the harness co-evolved without the VWM overall (59.22% vs. 58.62%), on MMSI-Cam. (42.57% vs. 35.34%) and on both SpatiaLab formats.

Together, these measurements support three points. At matched cost, majority voting plateaus, while CoWorld w/o VWM outperforms every sampling budget. Under the same limits, co-evolution improves the initial harness. Compared with evolution without VWM access, as in the evolving baselines, co-evolution with the VWM yields a stronger harness even when it generates no video. This indicates that the incremental value of co-evolution comes from VWM access during evolution, which also lets the harness run at low cost without it. We will add the initial-versus-final comparison and the co-evolution control to a new Appendix A.11 on the value of co-evolution, and the cost and majority-voting measurements to a new Appendix A.12 on inference cost and sampling baselines.

AuthorsOpenReview ↗

Response to Reviewer wM28 (3/3)

Q2: “Higher point estimates on every benchmark do not establish a reliable improvement across tasks. ... CausalSpatial's 28.1% recovery rate corresponds to correcting 154 baseline errors while losing 153 previously correct answers. ... [The authors] do not report paired uncertainty or variation across repeated runs. ... Can paired corrected/newly incorrect counts and uncertainty estimates from the same runs as Table 1, grouping repeated formats by original question, establish which benchmark gains are robust?”

Response: Thank you for this question and for working out the CausalSpatial counts. As you suggest, we report paired corrected and newly incorrect counts with uncertainty estimates from the same runs as Table 1, grouping repeated formats by original question, as well as three runs of both methods on MMSI and ERQA.

Paired tests on the Table 1 runs. We count how many direct-reasoning errors CoWorld corrects and how many correct answers it makes newly incorrect in the Table 1 runs. We compute paired 95% bootstrap intervals by resampling original questions, grouping SAT's two option orders and SpatiaLab's two formats, and obtain p-values from paired sign-flip tests:

BenchmarkDifference (pp) [95% CI]pCorrected / newly incorrect
MMSI+4.54 [+1.24, +7.73]0.010155 / 111
ERQA+6.75 [+2.00, +11.75]0.00964 / 37
Spin+1.51 [+0.07, +2.88]0.043208 / 167
SL-Open+1.43 [−0.64, +3.50]0.21122 / 102
SAT+1.33 [−2.33, +5.00]0.6116 / 12
SL-MCQ+0.71 [−1.50, +2.86]0.56124 / 114
PerceptionBench+0.51 [−1.11, +2.09]0.56297 / 282
CausalSpatial+0.07 [−2.37, +2.51]1.00154 / 153
All+1.40 [+0.62, +2.18]0.00051,140 / 978

As shown in the table, the overall gain and the gains on MMSI, ERQA and Spin are significant. Their paired intervals exclude zero, and the CausalSpatial counts (154 / 153) match your calculation.

Repeated runs on MMSI and ERQA. To test whether these gains replicate, we ran both direct reasoning and CoWorld two more times on MMSI and ERQA, for three runs including the Table 1 run. The table reports accuracy as mean ± SD over the three runs and the difference between the two methods:

BenchmarkDirect reasoningOursDifference (pp) [95% CI]p
MMSI39.00 ± 0.8842.54 ± 0.36+3.54 [+1.48, +5.57]0.001
ERQA54.33 ± 2.4058.33 ± 0.14+4.00 [+1.08, +7.00]0.012

The results indicate that the MMSI and ERQA gains replicate across runs. Both three-run differences are significant, and every CoWorld run outperforms every direct-reasoning run on both benchmarks.

These analyses indicate that the overall gain is significant, that the MMSI and ERQA gains hold in both the paired tests and the repeated runs, and that Spin is significant in the paired tests. Grouping repeated formats by original question, as you suggested, keeps these intervals from overstating the evidence on SAT and SpatiaLab. We will add the full paired table with intervals, p-values, and corrected and newly incorrect answers, along with the MMSI and ERQA three-run estimates, to a new Appendix A.10 on statistical tests and answer changes.


Summary

Thank you again for your constructive comments.

  1. To address your concerns, we (i) measured per-item inference cost, tokens and latency, and compared CoWorld with majority voting at matched cost; (ii) compared the initial and the final harness under the same limits; (iii) compared co-evolution with and without VWM access, which the five evolving baselines lack; and (iv) reported paired corrected and newly incorrect counts with uncertainty estimates from the Table 1 runs, grouping repeated formats by original question, along with three runs on MMSI and ERQA. We believe these measurements strengthen the benefit-cost justification and the evidence for the incremental value of co-evolution and for which gains are robust.
  2. If these responses have addressed your concerns, we would greatly appreciate knowing whether any other concerns affect your assessment. Your further feedback would be very helpful for improving the paper.
Reviewer_ZvKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoWorld Reasoner is an agentic framework in which a vision-language model (VLM) decides when to call a video world model (VWM) and how to use its outputs. The VWM supports viewpoint changes, temporal interpolation, sequence continuation, and simulated interventions. Using feedback from 120 development examples, a supervisor revises the tools, skills, and execution code while keeping both models’ weights fixed. The final harness is evaluated on seven benchmarks. Relative to direct reasoning with the same GPT-5.6 deployment, item-weighted accuracy increases from 58.29% to 59.69%, with the largest absolute gains on MMSI (+4.54 percentage points) and ERQA (+6.75 points).

Critical strengths

1- The ablation identifies where generated video helps. Figure 3 compares the full system with a version without the VWM. The gains on occlusion and collision reasoning, alongside a decline on OCR, show that the value of generated video depends on the task.

2-The evaluation tests transfer beyond the development benchmarks. SAT, SpatiaLab, and ERQA contributed no development examples. The final harness improves over direct reasoning on all three, with the largest gain on ERQA. This supports the claim that the workflow can transfer to benchmarks outside the development set.

Critical weaknesses

1- The evaluation does not establish the added value of co-evolution over a fixed VWM-enabled harness. The five evolving baselines use the same VLM but have no access to the video world model. As a result, these comparisons cannot distinguish the benefit of revising the harness from the benefit of adding a video tool and a multi-stage reasoning workflow. Figure 3’s no-VWM ablation helps assess the contribution of the video model, but it does not test whether evolving the tools, skills, and code improves an already functional VWM-enabled system. This is a substantive gap because co-evolution is presented as a central methodological contribution.

2- The evaluation gives an incomplete account of when generated video helps, hurts, or justifies its cost. The paper defines error recovery over examples the direct baseline answers incorrectly. This measures wrong-to-right changes but leaves right-to-wrong changes outside that analysis. Figure 3 reports a decline on OCR in the no-VWM comparison, making regressions worth examining directly. The overall gain is 1.40 percentage points, and gains vary substantially across benchmarks. A paired breakdown of corrections and regressions, especially on questions where a VWM call was made, would make it easier to judge the reliability of the added visual evidence. End-to-end latency and inference cost, including failed generations and retakes, are also needed to assess the practical trade-off.

Questions for authors

1- On the same held-out examples, how does the final co-evolved harness compare with the original fixed VWM-enabled harness under matched VLM, VWM, and test-time limits? Can you report per-benchmark accuracy and the number of model calls for both variants?

2- Compared with direct reasoning, how often does the final harness change an incorrect answer to a correct one, and how often does it change a correct answer to an incorrect one? Could you report these counts separately for questions with and without a VWM call, alongside paired uncertainty estimates and end-to-end latency and cost?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ZvKP (1/3)

Dear Reviewer ZvKP,

Thank you for the review and for noting the task-dependent video gains and transfer beyond the development benchmarks.

In this response, we use direct reasoning to denote the GPT-5.6 direct reasoning baseline, and w/o VWM to denote CoWorld with all VWM-driven tools and the verifier disabled at test time, keeping the skills and four-stage execution, as in the paper's "Ours (w/o VWM)".


Q1: “The evaluation does not establish the added value of co-evolution over a fixed VWM-enabled harness. ... these comparisons cannot distinguish the benefit of revising the harness from the benefit of adding a video tool and a multi-stage reasoning workflow. ... On the same held-out examples, how does the final co-evolved harness compare with the original fixed VWM-enabled harness under matched VLM, VWM, and test-time limits? Can you report per-benchmark accuracy and the number of model calls for both variants?”

Response: Thank you for proposing this comparison. We compare the fixed and the final harness on the same held-out items under matched limits, reporting per-benchmark accuracy and model calls for both. We additionally evolve the same initial harness without VWM access as a control, to separate harness revision from the video tool and the multi-stage workflow.

Comparison of the fixed and final harness. We use the initial harness of Algorithm 1 as the fixed harness: it already has the four VWM-driven tools, the verifier, the four-stage execution and the initial skills. We evaluated it and the final co-evolved harness on all 11,544 held-out items, with the same GPT-5.6 deployment, VWM and model-turn limits. The table reports per-benchmark accuracy and model calls per item; bold differences have paired 95% CIs that exclude zero:

BenchmarkFixed (%)Final (%)Difference (Final − Fixed, pp)Model calls/item (Fixed → Final)
MMSI37.9442.89+4.954.24 → 5.76
ERQA58.5058.50+0.004.12 → 4.51
CausalSpatial57.6360.79+3.154.21 → 5.03
Spin82.9183.76+0.854.16 → 4.69
SAT87.6789.67+2.004.06 → 4.91
SL-MCQ56.7157.36+0.644.09 → 4.33
SL-Open34.6437.64+3.004.14 → 4.32
PerceptionBench50.0051.35+1.354.10 → 4.12
All57.8659.69+1.844.14 → 4.58

As shown in the table, co-evolution improves an already functional VWM-enabled harness under matched limits. Overall accuracy rises from 57.86% to 59.69%, no benchmark declines, and the gains on MMSI, CausalSpatial and SL-Open are significant, as is the overall gain. Within the same limits, model calls per item rise from 4.14 to 4.58 on average.

AuthorsOpenReview ↗

Response to Reviewer ZvKP (2/3)

Q1, continued: Separating harness revision from the video tool and the multi-stage workflow. As a control, we also evolved the same initial harness without VWM access, using the same 120 development examples, rounds and revision budget. The table reports overall accuracy on the 11,544 held-out items for direct reasoning and for each harness, with the VWM disabled or enabled at test time as marked:

SettingVWM at test timeAccuracy (%)
Direct reasoningoff58.29
Initial harnessoff58.82
Co-evolved without the VWMoff58.62
Co-evolved with the VWMoff59.22
Co-evolved with the VWMon59.69

The results help separate the three factors. (1) Generated videos improve accuracy at test time. Enabling the VWM raises the accuracy of the harness co-evolved with the VWM from 59.22% to 59.69%. (2) Without generating videos, the harness still retains the visual reasoning capability acquired through co-evolution with the VWM. With the VWM disabled, the harness co-evolved with the VWM reaches 59.22%, compared with 58.82% for the initial harness and 58.62% for the harness co-evolved without the VWM, which does not improve on its starting point. The initial harness, with the VWM disabled, improves on direct reasoning from 58.29% to 58.82%.

Together, the two comparisons help separate the factors you raise. Under matched limits, co-evolution improves an already functional VWM-enabled harness. The control further shows that the initial harness improves on direct reasoning, that harness revision improves accuracy once the VWM is in the evolution loop, and that generated videos then add further accuracy at test time. We will add the fixed-versus-final table and the co-evolution control to a new Appendix A.11 on the value of co-evolution.


Q2: “End-to-end latency and inference cost, including failed generations and retakes, are also needed to assess the practical trade-off. ... Compared with direct reasoning, how often does the final harness change an incorrect answer to a correct one, and how often does it change a correct answer to an incorrect one? Could you report these counts separately for questions with and without a VWM call, alongside paired uncertainty estimates and end-to-end latency and cost?”

Response: Thank you for asking for these measurements. We report how often the final harness changes an incorrect answer to a correct one, and a correct answer to an incorrect one, relative to direct reasoning, separately for questions with and without a VWM call and with paired uncertainty estimates. We then report the end-to-end latency and cost, including failed generations and retakes.

Answer changes by VWM call. We compare the final harness with the Table 1 direct-reasoning run on all 11,544 held-out items. We compute paired 95% bootstrap CIs, resampling the two option orders of each SAT question and the two formats of each SpatiaLab question together:

VWM callItemsIncorrect → correctCorrect → incorrectAccuracy difference (pp) [95% CI]
Yes3,299463371+2.79 [+1.12, +4.50]
No8,245677607+0.85 [+0.00, +1.69]

As shown in the table, (1) corrections outnumber regressions in both groups. Incorrect-to-correct changes exceed correct-to-incorrect ones both on questions with a VWM call (463 vs. 371) and on those without (677 vs. 607). (2) The net gain is larger where the VWM is called. There, accuracy improves by 2.79 pp and the paired interval excludes zero, compared with 0.85 pp on questions without a call.

AuthorsOpenReview ↗

Response to Reviewer ZvKP (3/3)

Q2, continued: Latency and cost. We measured latency and cost on the 2,765 MMSI, ERQA and CausalSpatial items, for CoWorld with and without the VWM and, as additional sampling baselines, for Direct reasoning@k, a majority vote over k GPT-5.6 samples at the agent's medium reasoning effort, for k up to 15. Costs use the rates we paid and include the agent, verifier and video generation. We take mean end-to-end latency from the main evaluation runs; for Direct reasoning@k with k > 1, latency is not applicable (–), as it depends on how many samples run concurrently. The table reports macro accuracy, model calls, latency and cost per item:

MethodMacro accuracy (%)Model calls/itemMean latencyCost/item
Direct reasoning@152.42161 s$0.0008
Direct reasoning@352.683–$0.0024
Direct reasoning@553.075–$0.0040
Direct reasoning@752.737–$0.0056
Direct reasoning@953.179–$0.0073
Direct reasoning@1152.8411–$0.0091
Direct reasoning@1352.7813–$0.0108
Direct reasoning@1552.9715–$0.0126
Ours (w/o VWM)53.734.49280 s$0.0064
Ours54.065.21351 s$0.0662

The table supports two points. (1) Video generation accounts for most of CoWorld's cost. Disabling the VWM cuts cost per item by about 90%, to slightly below Direct reasoning@9, and none of the 3,299 videos generated in the main evaluation failed or required a retake. (2) CoWorld w/o VWM outperforms every direct-reasoning budget. Direct-reasoning accuracy fluctuates between 52.42% and 53.17% as the number of samples increases and reaches neither CoWorld configuration; the full system has the highest accuracy in the table.

These measurements show that the final harness makes more corrections than regressions, with the clearest net gain where it calls the VWM, and that its cost comes mainly from video generation, which ran without failures or retakes. CoWorld therefore offers two operating points: the full system for the highest accuracy, and CoWorld w/o VWM, which outperforms direct reasoning at every sampling budget at a lower cost than nine direct samples. We will add the answer-change table, split by VWM call, to a new Appendix A.10 on statistical tests and answer changes, and the latency and cost table to a new Appendix A.12 on inference cost and sampling baselines.


W2: “Figure 3 reports a decline on OCR in the no-VWM comparison, making regressions worth examining directly.”

Response: Thank you for pointing to the OCR decline. We examine this regression directly, first explaining why generated videos add little on OCR questions and then quantifying the change on these items.

OCR questions ask the agent to read text that is already visible in the input image. The VWM tools target spatial and temporal reasoning, so a generated view, transition or outcome adds no information about such text. We report accuracy on the 255 OCR items in the Table 1 runs:

MethodOCR accuracy (%)
Direct reasoning50.98
Ours (w/o VWM)50.59
Ours49.41

The VWM-enabled harness answers three fewer OCR items correctly than the same harness w/o VWM (−1.18 pp). To put this difference in context, we additionally look at PerceptionBench as a whole, which contains these OCR items: there, the final harness outperforms the mean of three direct-reasoning runs by +1.60 pp.

The OCR decline therefore reflects a task on which visual imagination supplies no new evidence, and it remains small within a benchmark where CoWorld improves overall. This matches the task-dependent pattern you noted: generated video helps on questions that require spatial or temporal reasoning, but it is not well suited to fine-grained perception tasks such as OCR.


Summary

Thank you again for your constructive comments.

  1. To address your concerns, we (i) compared the fixed and the final harness under matched limits, with a control evolved without the VWM; (ii) reported corrections and regressions relative to direct reasoning, separately for questions with and without a VWM call, with paired uncertainty estimates; (iii) measured end-to-end latency and cost, including failed generations and retakes; and (iv) examined the OCR decline directly. We believe these results strengthen the evidence for the added value of co-evolution over a fixed VWM-enabled harness and make its practical trade-off explicit.
  2. If these responses have addressed your concerns, we would greatly appreciate knowing whether any other concerns affect your assessment. Your further feedback would be very helpful for improving the paper.
Reviewer_zVrsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper enables vision-language models to generate videos—depicting new perspectives, motion sequences, or scene changes—on demand to assist in reasoning. It also incorporates a feedback mechanism to refine the tools and prompting processes used automatically. Performance improved across all seven evaluation datasets, with an overall accuracy gain of approximately 1.4 percentage points.

Critical strengths

The approach is flexible, allowing the model to select different forms of visual imagination tailored to specific problems. The experiments cover a diverse range of tasks, and the observed improvements on datasets not included in the optimization process suggest a degree of generalizability.

Critical weaknesses

The specific contribution of the video generation component remains somewhat ambiguous; some of the performance gains may stem primarily from improvements to the prompting and reasoning workflows. Furthermore, generated videos—while appearing plausible—may not accurately reflect reality, and the paper does not sufficiently address the risks posed by such potentially misleading content.

Questions for authors

When computational costs are comparable, what is the marginal benefit of generating videos versus simply refining the reasoning process? Has there been any analysis of the frequency with which the generated videos cause the model to arrive at an incorrect answer when it would otherwise have been correct?

4 comments
AuthorsOpenReview ↗

Response to Reviewer zVrs (1/3)

Dear Reviewer zVrs,

Thank you for the review and for highlighting the flexibility of our approach and its potential to generalize.

In this response, we use direct reasoning to denote the GPT-5.6 direct reasoning baseline, and w/o VWM to denote CoWorld with all VWM-driven tools and the verifier disabled at test time, keeping the skills and four-stage execution, as in the paper's "Ours (w/o VWM)".


W1: “The specific contribution of the video generation component remains somewhat ambiguous; some of the performance gains may stem primarily from improvements to the prompting and reasoning workflows.”

Response: Thank you for raising the question of where the gains come from. To separate the contribution of video generation from improvements to prompting and the reasoning workflow, we conducted an additional controlled comparison that examines two points: first, whether generated videos improve accuracy at test time; second, whether the harness retains the visual reasoning capability acquired through co-evolution with the VWM even when it generates no video.

We evolved the same initial harness with and without VWM access, using the same 120 development examples, rounds and revision budget, and evaluated each harness on all 11,544 held-out items with the VWM disabled at test time (w/o VWM); the harness co-evolved with the VWM was also evaluated with the VWM enabled. Besides overall accuracy, we report the 249 MMSI positional-relationship questions involving the camera (Cam.–Cam., Cam.–Obj. and Cam.–Reg.; MMSI-Cam.), the two SpatiaLab formats, which provided no development examples, and CausalSpatial, a benchmark of reasoning under object interventions. Each entry is the accuracy on the full benchmark or subset:

Harness (VWM at test time)Overall accuracy (%)MMSI-Cam. (%)SL-MCQ (%)SL-Open (%)CausalSpatial (%)
Initial harness (off)58.8235.7456.9334.5759.71
Co-evolved without VWM (off)58.6235.3454.6434.3660.43
Co-evolved with VWM (off)59.2242.5756.8636.5058.49
Co-evolved with VWM (on)59.6947.7957.3637.6460.79

As shown in the table, (1) generated videos improve accuracy at test time. For the harness co-evolved with the VWM, enabling the VWM improves accuracy in every column: overall from 59.22% to 59.69%, on MMSI-Cam. from 42.57% to 47.79% and on CausalSpatial from 58.49% to 60.79%, and also on both SpatiaLab formats. On MMSI-Cam. and CausalSpatial, the agent requests a new view and an intervention outcome as additional visual evidence, respectively. (2) Without generating videos, the harness still retains the visual reasoning capability acquired through co-evolution with the VWM. With the VWM disabled, the harness co-evolved with the VWM outperforms the harness co-evolved without it overall (59.22% vs. 58.62%), on MMSI-Cam. (42.57% vs. 35.34%) and on both SpatiaLab formats. It also outperforms the initial harness overall, on MMSI-Cam. and on SL-Open. In contrast, the harness co-evolved without the VWM does not improve on the initial harness overall (58.62% vs. 58.82%).

Together, these results help isolate the specific contribution of video generation. Beyond improvements to prompting and the reasoning workflow, generated videos improve accuracy at test time, and co-evolution with the VWM gives the harness a visual reasoning capability that it retains even without generating videos. The two roles complement each other: the harness is revised with the VWM during co-evolution and still calls it where a question needs new visual evidence. We will state these two roles explicitly in Sec. 3.2 VLM-VWM Harness Co-Evolution.

AuthorsOpenReview ↗

Response to Reviewer zVrs (2/3)

W2: “generated videos—while appearing plausible—may not accurately reflect reality, and the paper does not sufficiently address the risks posed by such potentially misleading content.”

Response: Thank you for raising the question about the reliability of the generated videos. We first report an additional evaluation of whether the generated videos faithfully realize their requests, and then describe how our harness checks video evidence before the agent answers.

The majority of generated videos are faithful to their requests. We had three external VLMs independently rate all 3,299 generated videos. We count a video as faithful only when the requested transformation or action appears, the target keeps its identity and the necessary scene landmarks remain. The table reports the share of videos each VLM judges faithful:

External VLMFaithful to request (%)
GLM-5.3-Flash96.15
Claude Haiku 5.588.54
Qwen 3.7 Flash83.60

All three VLMs judge the majority of videos faithful, from 83.60% to 96.15%.

Our harness checks video evidence with a verifier. To check generated content before it enters the answer, we use a verifier that reports visible artifacts and uncertainty in each video to the agent. The final stage then asks the agent to state the question's reference frame and the evidence that distinguishes the answer options before it answers. The agent is therefore asked to interpret each generated video against the original question rather than accept it at face value.

Together, these two aspects show that most generated videos are faithful to their requests, and that the harness treats each video as evidence that needs to be checked before it enters the answer; Q2 below further measures how often a correct answer becomes incorrect after the video enters. We will add the rating criteria and the per-VLM faithfulness results to a new Appendix A.13 on the evaluation of generated videos, and describe in Sec. 3.1 Video World Model-Driven Harness how the verifier reports artifacts and uncertainty to the agent before it answers.


Q1: “When computational costs are comparable, what is the marginal benefit of generating videos versus simply refining the reasoning process?”

Response: Thank you for suggesting this comparison. To measure the marginal benefit of generating videos over refining the reasoning, we compare CoWorld with majority voting over direct-reasoning samples in two computation-matched settings: first at matched model calls, and then at matched cost.

Comparison at matched model calls. We compare CoWorld with Direct reasoning@k, a majority vote over k GPT-5.6 samples at the same medium reasoning effort as the agent. On the 2,765 MMSI, ERQA and CausalSpatial items, CoWorld averages 5.21 model calls per item, between the budgets @5 and @9. The table reports accuracy on these items, with pooled accuracy and MMSI-Cam. results in the last two rows:

BenchmarkOurs (5.21 calls/item)@1@5@9
MMSI42.8940.1040.1040.21
ERQA58.5056.0056.7557.00
CausalSpatial60.7961.1562.3762.29
Pooled54.1853.0253.7453.78
MMSI-Cam.47.7935.7436.9538.15

As shown in the table, (1) additional samples change direct-reasoning accuracy little. On MMSI, direct-reasoning accuracy stays at 40.10–40.21% from @1 to @9. (2) CoWorld outperforms majority voting at matched calls. Pooled over the three benchmarks, CoWorld outperforms both @5 and @9, and the gap is largest on MMSI-Cam., where CoWorld reaches 47.79% compared with 38.15% for @9. On CausalSpatial, majority voting is slightly higher. Additional samples reason over the same input images, whereas CoWorld can request the visual evidence a question needs.

AuthorsOpenReview ↗

Response to Reviewer zVrs (3/3)

Q1, continued: Comparison at matched cost. We further extend the comparison to 15 samples on the same items and measure cost per item at the rates we paid, including video generation for CoWorld. Accuracy in this table is the equal-weighted average over the three benchmarks (macro accuracy), whereas the pooled row above weights items equally:

MethodMacro accuracy (%)Model calls/itemCost/item
Direct reasoning@152.421$0.0008
Direct reasoning@352.683$0.0024
Direct reasoning@553.075$0.0040
Direct reasoning@752.737$0.0056
Direct reasoning@953.179$0.0073
Direct reasoning@1152.8411$0.0091
Direct reasoning@1352.7813$0.0108
Direct reasoning@1552.9715$0.0126
Ours (w/o VWM)53.734.49$0.0064
Ours54.065.21$0.0662

As shown in the table, (1) the accuracy of direct reasoning plateaus as its sampling cost increases. From @1 to @15, macro accuracy rises by only 0.55 pp, with fluctuations between budgets. (2) CoWorld w/o VWM outperforms every sampling budget. At about the cost of @9, it reaches 53.73%, and enabling the VWM further improves macro accuracy to 54.06%, the highest in the table.

Across both settings, refining the reasoning through more samples of the same input soon plateaus. The harness co-evolved with the VWM outperforms every sampling budget even without generating a video, and generated videos add further accuracy on top of it. This indicates that the marginal benefit of the VWM lies both in what the harness acquires through co-evolution with it and in the visual evidence it provides at test time, which more samples of the same input cannot supply. We will add the majority-voting results for 1 to 15 samples and the per-item cost table to a new Appendix A.12 on inference cost and sampling baselines.


Q2: “Has there been any analysis of the frequency with which the generated videos cause the model to arrive at an incorrect answer when it would otherwise have been correct?”

Response: Thank you for this question. We measure how often CoWorld turns an initially correct answer into an incorrect one on the items where it calls the VWM. As answers can also change between runs without any video, we additionally compare this rate with the changes between three independent direct-reasoning runs on the same items.

CoWorld starts from an initial answer by direct reasoning under the initial prompt, before any video is generated. On the 3,299 items where it calls the VWM, we compare its final answer with this initial answer and with the Table 1 direct-reasoning run (run 1 below). The table reports the share of items changing from correct to incorrect:

RunsCorrect → incorrect (% of the 3,299 items)
Direct reasoning run 1 → run 29.7
Direct reasoning run 1 → run 38.6
Direct reasoning run 2 → run 39.2
Ours, same run (initial answer → final answer)7.7
Direct reasoning run 1 → Ours11.2

As shown in the table, (1) correct answers also become incorrect across direct-reasoning runs. Between two runs, 8.6–9.7% of the same items change from correct to incorrect. (2) After the video enters, a correct initial answer becomes incorrect on 7.7% of the items. (3) Relative to direct reasoning, CoWorld's corrections outnumber its regressions. Against run 1, CoWorld changes 371 items (11.2%) from correct to incorrect and corrects 463 errors, a net gain of 2.79 pp.

These comparisons show that a correct answer turning incorrect is an ordinary part of LLM/VLM inference, and that on the items where CoWorld calls the VWM, its corrections outnumber such changes. We will add these answer-change comparisons to a new Appendix A.10 on statistical tests and answer changes.


Summary

Thank you again for your constructive comments.

  1. To address your concerns, we (i) conducted an additional controlled comparison of the role of generated videos at test time and during co-evolution; (ii) evaluated the faithfulness of all 3,299 generated videos with three external VLMs and described how the verifier checks video evidence; (iii) compared CoWorld with majority voting at matched calls and cost; and (iv) measured how often a correct answer becomes incorrect after the video enters, against direct-reasoning re-runs. We believe these results strengthen the evidence for the contribution of video generation and the reliability of the generated videos.
  2. We would be grateful for any further feedback, which would be very helpful for improving the paper.

Dear Reviewer zVrs,

Thank you again for your positive feedback and for your time. We would greatly appreciate any further comments or suggestions, and would be glad to continue the discussion while there is still time to respond.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.