acceptodds
Under review as a conference paper at ICLR 2027

AgiWorld: An Agentic Video World Model for Building Persistent Worlds

Abstract

Recent interactive video world models generate visually compelling videos from a single image, following user-provided camera motion and text prompts. However, such low-level control demands substantial user effort to realize a high-level intent, and the generated world struggles to remain persistent as the exploration proceeds. In this work, we present AgiWorld, an agentic video world model that explores and builds a persistent world from a single image and a free-form instruction. For high-level planning, our agent AgiWorld-Agent designs the world and the journey through it in a world state and directs the exploration. For low-level realization, our video world model AgiWorld-WM generates the video chunk by chunk and stores it with its geometry in a spatial memory, from which the agent updates its world state. This closed loop keeps the world faithful to the user's intent and keeps the agent's world state consistent with the generated content throughout the exploration. We co-design the two levels around the shared spatial memory and an interface of metric camera trajectories and structured prompts. In AgiWorld-WM, to handle occlusion and accumulated geometric errors in the spatial memory over long rollouts, we propose occlusion-aware history selection, which retrieves past views by surface visibility, and reliability-based memory maintenance, which prefers reliable geometry and earlier observations. AgiWorld-WM achieves state-of-the-art overall performance on world revisiting and exploration benchmarks, and qualitative results and ablations show that AgiWorld builds persistent worlds from free-form instructions over minute-long journeys. Persistence thus arises not from the video model alone but from its closed loop with the agent.

Primary area:
generative models
Keywords:
camera-controlled video generation, long video generation, world model, agent system

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FoT9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AgiWorld, a system that builds and explores a persistent world from a single image and a free-form instruction. It combines AgiWorld-Agent, which maintains a high-level world state and plans the journey, with AgiWorld-WM, an autoregressive video model equipped with spatial memory. The world model introduces occlusion-aware history selection and reliability-based memory maintenance. Experiments on DL3DV-Evaluation and the proposed World Explore benchmark show good video quality and camera-following results, while qualitative examples illustrate the complete system.

Critical strengths
  1. The division between a semantic world state maintained by the agent and a geometric memory maintained by the video model is sensible. The two components have a clear interface through metric camera trajectories, prompts, and generated geometry.

  2. The world-model component is technically strong. AgiWorld-WM performs well across visual quality, camera following, and reprojection metrics, and the proposed memory mechanisms address concrete long-rollout problems: occlusion and accumulated geometric error.

Critical weaknesses
  1. The complete agentic system is evaluated mainly through a few qualitative examples. Most quantitative experiments evaluate AgiWorld-WM under given trajectories and prompts, rather than the central claim that the closed-loop agent can reliably convert high-level instructions into persistent worlds. The main evaluation contains only a small number of complete-system examples and no aggregate instruction-completion or semantic-consistency results.

  2. Although the evaluation includes revisiting trajectories, the metrics do not directly measure consistency between an initial visit and a later revisit. FID and short-interval depth reprojection are useful but do not establish that a place remains the same after a long absence. The system can also regenerate failed chunks, but the observed retry frequency and computational overhead are not reported, making it hard to separate state-update gains from additional sampling.

Questions for authors
  1. Can the authors report aggregate full-system results over a reasonably sized instruction set, including completion rate, realization of requested objects, long-range revisitation consistency, and failure rate, including quantitative evaluation of the existing agent ablations?

  2. How often does chunk regeneration occur? Under an equal generation budget, how does the closed-loop system compare with an open-loop planner plus simple resampling?

  3. Can the authors directly compare a region at its initial visit and a later revisit, grouped by the elapsed time or camera travel distance between visits?

5 comments
AuthorsOpenReview ↗

Q1: Quantitative Evaluation of AgiWorld-Agent

1. Evaluation protocol

To evaluate the complete agentic system, we construct an instruction-following test set and an evaluation protocol that measures instruction completion, world consistency and geometric failures. The test set has 20 image–instruction pairs of indoor and outdoor scenes. Each instruction asks for a route and names objects that should appear along it, e.g., "walk out through the open garage door and down the driveway, passing the silver car on your left." We evaluate the full system and the four agent ablations of Figure 9. All variants use the same input image, instruction, world model, and initial seed. Each ablation is capped at the number of output chunks produced by the full system.

Metrics.

  • Instruction completion: Route and Objects. Before evaluation, each instruction was split, from its text alone, into a fixed checklist of route steps and requested objects. Each item is scored done (1), partial (0.5) or not done (0), and we report the mean over the route steps and over the objects.
  • World consistency: Structure, Content and Appearance. Each video is rated from 1 to 5 on whether the 3D structure and layout, the objects in the scene, and the lighting and materials stay consistent.
  • Geometric failures. We apply the same geometric checks in Appendix A.6 to the final output of each chunk after the first and report the percentage of chunks that fail at least one check.

Judges. Route, Objects, Structure, Content and Appearance are scored by MLLM judges. Specifically, we use two of the strongest available MLLMs, Claude Opus 5.5 (xhigh effort) and Claude Fable 5.2 (medium effort), as the two judges; both differ from the MLLM used by our agent (GPT-6-Astra). Geometric failures are computed directly from the generated geometry.

For each metric, the judges receive only the input image, video frames sampled at 2 fps, and the corresponding evaluation question in a separate call. For Route and Objects, the question includes the fixed checklist derived from the instruction. Scores are averaged over the two judges and then over instructions. The two judges give the same answer on 94% of route steps and 92% of objects, and 97% of their consistency ratings differ by at most one point.

2. Results

Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑) and geometric failures (% of chunks ↓)

Route ↑Objects ↑Structure ↑Content ↑Appearance ↑Geometric failures ↓
Full system1.000.933.403.053.200%
w/o agent0.880.792.552.302.4521%
w/o world state0.960.933.382.832.8811%
w/o state update0.960.863.002.953.008%
w/o geometric grounding0.990.883.353.102.9514%

The full system is first in every column except content consistency, where it is second. Compared with the full system:

  • w/o agent plans the whole path once and uses the instruction as every prompt. It is the worst on every metric, by a wide margin: route completion drops to 0.88, objects to 0.79, consistency by 0.75 to 0.85, and geometric failures rise to 21%.
  • w/o world state plans each chunk only from what has been generated so far. Its route and object completion stay close to the full system, but its content and appearance consistency fall to the lowest after w/o agent (3.05 → 2.83, 3.20 → 2.88), and geometric failures rise to 11%.
  • w/o state update generates the whole video from the initial plan, without review. Structure consistency drops (3.40 → 3.00), and geometric failures rise to 8%.
  • w/o geometric grounding uses neither the point cloud nor the geometric tools. Geometric failures rise from 0% to 14%.

The evaluation above measures instruction completion and world consistency under agent-generated journeys. Separately, our response to Q3 evaluates long-range revisit consistency of AgiWorld-WM under common prescribed trajectories, grouping revisit pairs by elapsed time and camera travel distance.

AuthorsOpenReview ↗

Q2: Regeneration overhead and comparison with open-loop resampling

We added the requested open-loop + resampling comparison on the same 20 image–instruction pairs used in our response to Q1.

The baseline keeps its initial plan fixed, as in w/o state update, and performs no MLLM review or state updates. When a chunk fails the same camera checks used by the full system (Appendix A.6), it regenerates the chunk using the same camera trajectory and prompt with a new seed. It uses the same regeneration limits as the full system.

The full system regenerates 6% of its chunks, corresponding to 1.06 world-model generations per output chunk. Open loop + resampling averages 1.03 under the same regeneration limits.

Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑), geometric failures (% of chunks ↓), world-model generations per chunk, and LLM tokens per 60-s rollout

Route ↑Objects ↑Structure ↑Content ↑Appearance ↑Geometric failures ↓WM generation / chunkTokens (in / out)
Full system1.000.933.403.053.200%1.061.34M / 0.27M
open loop (w/o state update)0.960.863.002.953.008%1.000.10M / 0.03M
open loop + resampling0.980.863.102.953.055%1.030.10M / 0.03M

Adding resampling reduces geometric failures from 8% to 5%, but leaves the Objects and Content scores unchanged. The full system achieves an Objects score of 0.93, higher consistency scores across all three dimensions, and a geometric-failure rate of 0%, using 1.06 world-model generations per output chunk compared with 1.03 for open loop + resampling.

The full system uses more LLM tokens to review generated observations, update its world state, and replan using the generated geometry. We report this additional agent cost explicitly. Under the same regeneration limits, these results support the benefit of closed-loop review and replanning over resampling a fixed initial plan.

AuthorsOpenReview ↗

W2 / Q3: Revisit consistency grouped by elapsed time and camera travel distance

We agree that FID and short-interval depth reprojection do not show that a place stays the same after a long absence. We therefore added an evaluation that directly compares two generated views of the same place: one from the earlier visit and one from the delayed revisit. This evaluates whether previously generated scene content is preserved when the camera returns after a delay.

We select revisit pairs on DL3DV-Evaluation, whose trajectories return to where they started. A frame at 15 s or later forms a revisit pair with an earlier frame, generated 15 to 60 seconds before, if their requested cameras are within 1.5 m in position and 60° in viewing direction. If several earlier frames qualify, we choose the one with the smallest camera-position distance. Frames without a qualifying earlier view are excluded. We warp the earlier frame into the revisit view and compute LPIPS between the two generated frames on the non-sky pixels visible in both. All methods are scored on the same pairs, using their Table 1 rollouts. For the distance-based analysis, we group the same revisit pairs by the cumulative distance traveled along the camera trajectory between the two visits.

Revisit consistency (LPIPS ↓, grouped by elapsed time)

15–30 s30–45 s45–60 sAll
SPMem.421.449.458.440
GEN3C.326.348.318.331
HY-WorldPlay.520.586.607.566
LingBot-World.500.537.539.523
Matrix-Game 3.0.450.468.466.460
Lyra 2.0.277.320.316.302
SANA-WM.321.340.344.333
AgiWorld-WM (ours).277.298.296.289

Revisit consistency (LPIPS ↓, grouped by camera travel)

LPIPS ↓5–15 m15–25 m≥25 m
SPMem.436.457.409
GEN3C.328.342.312
HY-WorldPlay.544.590.577
LingBot-World.519.538.495
Matrix-Game 3.0.467.466.426
Lyra 2.0.276.321.328
SANA-WM.328.341.325
AgiWorld-WM (ours).275.302.282

AgiWorld-WM matches Lyra 2.0 at 15–30 s and achieves the lowest revisit error at 30–45 s, 45–60 s, and overall. These results provide direct evidence that our method better preserves previously generated content across longer revisit delays. Grouped by camera travel, AgiWorld-WM has the lowest error in every range, and its margin over the strongest baseline grows with the distance traveled: .001 at 5–15 m, .019 at 15–25 m and .030 at ≥25 m.

Reviewer_9D4pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AgiWorld, an agentic video world model that turns an image and a free-form instruction into a camera-guided journey through a generated static world. Its agent maintains a layout of observed and imagined elements, plans camera trajectories and prompts, and revises the plan after inspecting each generated chunk. The video model uses spatial memory, with surface-aware history retrieval and reliability-based maintenance to reduce inconsistencies. Quantitative comparisons and memory ablations evaluate the video model on 60-second revisiting and exploration sequences, while demonstrations and qualitative agent ablations illustrate the complete system.

Critical strengths

The memory design addresses concrete problems in geometry-guided generation: occluded surfaces entering the retrieved context and inconsistent geometry accumulating across chunks. The comparisons against seven baselines and the memory ablations provide evidence that the complete design improves generation quality and geometric consistency, including in later portions of the rollout. This is the strongest empirical contribution.

The agent maintains an explicit distinction between observed content and planned content, and connects both to metric camera planning and structured prompts. Updating this state from generated observations provides a coherent approach to coordinating a longer journey. The detailed implementation and qualitative ablations make this design useful for further work on instruction-guided world generation.

Critical weaknesses

The central claim about autonomous, instruction-faithful world building is supported mainly by demonstrations. Figure 9 presents agent ablations on two examples, without aggregate completion rates, consistency measurements, or variability across runs. The system also reviews and regenerates chunks, but the contribution and cost of this additional sampling are not quantified. The video-model results are useful, yet they do not establish how reliably the complete system follows diverse instructions or how much each agent component contributes.

Questions for authors

Could you evaluate the full agent and its ablations across a defined set of instructions, reporting completion rates and unresolved instruction or geometry failures? Please also report regeneration frequency and cost, and clarify whether the variants receive comparable generation budgets.

Do the improvements hold when consistency is measured directly between earlier observations and delayed revisits of the same region? Results across different revisit delays would help distinguish persistent memory from improved local quality.

2 comments
AuthorsOpenReview ↗

Q1: Quantitative Evaluation of AgiWorld-Agent

1. Evaluation protocol

To evaluate the complete agentic system, we construct an instruction-following test set and an evaluation protocol that measures instruction completion, world consistency and geometric failures. The test set has 20 image–instruction pairs of indoor and outdoor scenes. Each instruction asks for a route and names objects that should appear along it. We evaluate the full system and the four agent ablations of Figure 9. All variants use the same input image, instruction, world model, and initial seed. Each ablation is capped at the number of output chunks produced by the full system. All metrics are computed on the final retained outputs after any regeneration. Partially fulfilled and unmet instruction items remain included in the Route and Objects scores.

Metrics.

  • Instruction completion: Route and Objects. Before evaluation, each instruction was split, from its text alone, into a fixed checklist of route steps and requested objects. Each item is scored done (1), partial (0.5) or not done (0), and we report the mean over the route steps and over the objects.
  • World consistency: Structure, Content and Appearance. Each video is rated from 1 to 5 on whether the 3D structure and layout, the objects in the scene, and the lighting and materials stay consistent.
  • Geometric failures. We apply the same geometric checks in Appendix A.6 to the final output of each chunk after the first and report the percentage of chunks that fail at least one check.

Judges. Route, Objects, Structure, Content and Appearance are scored by MLLM judges. Specifically, we use two of the strongest available MLLMs, Claude Opus 5.5 (xhigh effort) and Claude Fable 5.2 (medium effort), as the two judges; both differ from the MLLM used by our agent (GPT-6-Astra). Geometric failures are computed directly from the generated geometry.

For each metric, the judges receive only the input image, video frames sampled at 2 fps, and the corresponding evaluation question in a separate call. For Route and Objects, the question includes the fixed checklist derived from the instruction. Scores are averaged over the two judges and then over instructions. The two judges give the same answer on 94% of route steps and 92% of objects, and 97% of their consistency ratings differ by at most one point.

2. Results

Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑), geometric failures (% of chunks ↓), world-model generation per chunk, and LLM tokens per 60-s rollout

Route ↑Objects ↑Structure ↑Content ↑Appearance ↑Geometric failures ↓WM generation / chunkTokens (in / out)
Full system1.000.933.403.053.200%1.061.34M / 0.27M
w/o agent0.880.792.552.302.4521%1.001.7K / 1.1K
w/o world state0.960.933.382.832.8811%1.000.08M / 0.03M
w/o state update0.960.863.002.953.008%1.000.10M / 0.03M
w/o geometric grounding0.990.883.353.102.9514%1.080.65M / 0.15M

Compared with the full system:

  • w/o agent plans the whole path once and uses the instruction as every prompt. It performs worst on all six evaluation metrics, with Route and Objects scores of 0.88 and 0.79, consistency scores lower by 0.75–0.85, and a geometric-failure rate of 21%.
  • w/o world state plans each chunk only from what has been generated so far. Its route and object completion stay close to the full system, but its content and appearance consistency fall to the lowest after w/o agent (3.05 → 2.83, 3.20 → 2.88), and geometric failures rise to 11%.
  • w/o state update generates the whole video from the initial plan, without review. Structure consistency drops (3.40 → 3.00), and geometric failures rise to 8%.
  • w/o geometric grounding uses neither the point cloud nor the geometric tools. Geometric failures rise from 0% to 14%.

3. Regeneration and generation budgets

The full system regenerates 6% of its chunks and w/o geometric grounding 8%, while the other variants do not regenerate. Including these attempts, all variants use 1.00–1.08 world-model generations per output chunk. The full system uses approximately 6% more generation calls per output chunk than the variants without regeneration. W/o geometric grounding uses more generation calls than the full system, yet its geometric-failure rate remains higher at 14%.

The full system uses more LLM tokens for repeated planning, review, and world-state updates, which process generated frames, geometry visualizations, and camera measurements. W/o agent and w/o state update plan once without review. W/o world state makes one call per chunk using the input image, latest frames and current pose, while w/o geometric grounding omits geometry inputs.

AuthorsOpenReview ↗

Q2: Consistency across different revisit delays

We added an evaluation that directly compares two generated views of the same place: one from the earlier visit and one from the delayed revisit. This evaluates whether previously generated scene content is preserved when the camera returns after a delay.

We select revisit pairs on DL3DV-Evaluation, whose trajectories return to where they started. A frame at 15 s or later forms a revisit pair with an earlier frame, generated 15 to 60 seconds before, if their requested cameras are within 1.5 m in position and 60° in viewing direction. If several earlier frames qualify, we choose the one with the smallest camera-position distance. Frames without a qualifying earlier view are excluded. We warp the earlier frame into the revisit view and compute LPIPS between the two generated frames on the non-sky pixels visible in both. All methods are scored on the same pairs, using their Table 1 rollouts.

Revisit consistency (LPIPS ↓, by delay between the two visits)

15–30 s30–45 s45–60 sAll
SPMem.421.449.458.440
GEN3C.326.348.318.331
HY-WorldPlay.520.586.607.566
LingBot-World.500.537.539.523
Matrix-Game 3.0.450.468.466.460
Lyra 2.0.277.320.316.302
SANA-WM.321.340.344.333
AgiWorld-WM (ours).277.298.296.289

AgiWorld-WM matches Lyra 2.0 at 15–30 s and achieves the lowest revisit error at 30–45 s, 45–60 s, and overall. These results provide direct evidence that our method better preserves previously generated content across longer revisit delays.

Reviewer_7JVjOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgiWorld, an agentic framework that generates minute-long, three-dimensional consistent videos from a single input image and a free-form natural language instruction. It couples a high-level planning agent (AgiWorld-Agent), which uses a multimodal language model and deterministic geometric tools to maintain a world state and plan metric camera trajectories and structured prompts, with a camera-conditioned autoregressive video diffusion model (AgiWorld-WM) in a closed loop. To prevent geometric drift and occlusion artifacts over long rollouts, the video world model incorporates surfel-based occlusion-aware history selection and free-space reliability-based memory maintenance. Quantitative evaluations on sixty-second revisiting and exploration benchmarks show that AgiWorld-WM outperforms existing camera-controlled baselines in visual quality and geometric consistency, while qualitative ablations demonstrate the effectiveness of the closed-loop agent design.

Critical strengths
  1. Closed-Loop Co-Design Between Symbolic Planning and Metric Video Generation:

Grounding both AgiWorld-Agent and AgiWorld-WM in a shared metric coordinate frame anchored at the initial image effectively bridges high-level semantic intent and chunk-level trajectory and prompt control. Offloading 3D computations (such as visibility rasterization, C1 waypoint interpolation, and collision verification) to deterministic geometric tools while using the multimodal language model to reconcile confirmed and imagined layout elements prevents spatial hallucinations that break open-loop planners over minute-long rollouts.

  1. Spatial Memory Mechanisms for Long-Horizon Geometric Consistency:

Representing depth-lifted memory points as oriented surfels for greedy set-cover retrieval (occlusion-aware history selection) and pruning later drifted points that violate earlier observed free space (reliability-based memory maintenance) directly addresses occluded-surface contamination and compounding depth drift. These parameter-free geometric mechanisms yield strong quantitative results on 60-second trajectories across DL3DV-Evaluation and World Explore, with ablations confirming substantial error reductions in the final 10 to 30 seconds of generation.

Critical weaknesses
  1. Lack of Quantitative Validation and Baseline Comparisons for the Full Agentic Pipeline:

Although the paper claims that world persistence arises from the closed loop between AgiWorld-Agent and AgiWorld-WM, all quantitative experiments evaluate only AgiWorld-WM on fixed ground-truth trajectories. The full system and agent ablations are evaluated strictly qualitatively on two to three scenes, without quantitative metrics (such as instruction completion rate, semantic consistency, or collision and failure rates across a test set), trigger statistics for review regeneration and replanning, or comparisons against prior agentic world generators (such as WonderJourney, WonderVerse, and LingBot-World 2.0). This leaves the reliability and generalizability of AgiWorld-Agent unsubstantiated.

  1. Estimator Coupling in Geometric Evaluation and Unexamined Overhead of the Heuristic Agent Pipeline:

First, Depth Anything 3 is used to annotate training poses and depths for AgiWorld-WM, estimate inference geometry, and compute test-time geometric errors (Trans., Rot., and Repr.), creating a favorable evaluation bias toward AgiWorld-WM over baselines trained with different pose estimators. Moreover, conditioning inference depth estimation on the requested poses rather than realized poses risks corrupting the spatial memory when camera following drifts. Second, AgiWorld-Agent relies on six post-chunk step types (Review, Revision, Refresh, Trajectory, up to three rounds of Fix, and Prompt), up to four chunk regenerations per rollout, and dozens of manually tuned thresholds, yet the paper reports no inference latency, token cost, or sensitivity analysis to these heuristics.

  1. Omission and Inadequate Attribution of Closely Related Literature on Memory-Augmented World Models:

The paper's positioning of its spatial memory modules omits or under-attributes key prior works. First, the related work section characterizes history-compression methods (such as FramePack, Zhang et al., 2025) as purely temporal and separate from spatial retrieval, omitting WorldPack (Oshima et al., 2025), which already combines compressed temporal trajectory packing with camera-guided spatial memory retrieval within a fixed token budget for long-horizon loop closure. This hybrid memory paradigm is closely mirrored by AgiWorld-WM's temporal-plus-spatial history context. Other prominent memory architectures for long-horizon video world models, including Long-Context State-Space Video World Models (Po et al., 2025), SlowFast-VGen (Hong et al., 2025), StarGen (Zhai et al., 2025), and History-Guided Video Diffusion (Song et al., 2025), are also omitted. Second, while VMem (Li et al., 2025d) is cited in passing in the introduction and related work, it is omitted from the method description of occlusion-aware history selection, where the authors present surfel-based normal and occlusion testing for history retrieval as a novel fix to point-cloud rendering without clarifying how their formulation differs from VMem's surfel-indexed view memory or comparing against it.

Questions for authors
  1. Quantitative Evaluation of AgiWorld-Agent:

Can you provide quantitative results for the full AgiWorld loop and the agent ablations (w/o agent, w/o world state, w/o state update, and w/o geometric grounding) across a diverse benchmark of scenes and instructions, reporting instruction completion rates, camera-measurement failure rates, and the average trigger frequencies of chunk regeneration, layout revision, and route replanning?

  1. Computational Overhead and Backbone Sensitivity:

What are the average wall-clock runtime, token consumption, and number of language model and diffusion calls required to generate a 60-second rollout, and how does AgiWorld-Agent perform when substituting GPT-6-Astra with smaller or open-weight multimodal language models?

  1. Independent Geometric Evaluation and Pose Drift Handling:

Do the camera-following and reprojection gains hold when evaluated with an independent structure-from-motion or multi-view estimator (such as COLMAP or MASt3R) instead of Depth Anything 3? Furthermore, when AgiWorld-WM exhibits 1.05 m average translation drift on World Explore, how does storing the requested poses rather than estimated poses affect memory alignment and the agent's ground and collision measurements?

  1. Distinction from Prior Memory-Augmented World Models (WorldPack and VMem):

How does AgiWorld-WM's hybrid temporal and spatial history context conceptually and empirically compare to WorldPack (Oshima et al., 2025), which similarly pairs recent trajectory packing with geometry-based memory retrieval under a fixed token budget? Additionally, could you clarify the exact technical differences between your surfel-based occlusion-aware history selection and the surfel-indexed view memory in VMem (Li et al., 2025d), and include both methods in the discussion and comparisons?

7 comments
AuthorsOpenReview ↗

W1 / Q1: Quantitative Evaluation of AgiWorld-Agent

1. Evaluation protocol

To evaluate the complete agentic system, we construct an instruction-following test set and an evaluation protocol that measures instruction completion, world consistency and geometric failures. The test set has 20 image–instruction pairs of indoor and outdoor scenes. Each instruction asks for a route and names objects that should appear along it, e.g., "walk out through the open garage door and down the driveway, passing the silver car on your left." We evaluate the full system and the four agent ablations of Figure 9. All variants use the same input image, instruction, world model, and initial seed. Each ablation is capped at the number of output chunks produced by the full system.

Metrics.

  • Instruction completion: Route and Objects. Before evaluation, each instruction was split, from its text alone, into a fixed checklist of route steps and requested objects. Each item is scored done (1), partial (0.5) or not done (0), and we report the mean over the route steps and over the objects.
  • World consistency: Structure, Content and Appearance. Each video is rated from 1 to 5 on whether the 3D structure and layout, the objects in the scene, and the lighting and materials stay consistent.
  • Geometric failures. We apply the same geometric checks in Appendix A.6 to the final output of each chunk after the first and report the percentage of chunks that fail at least one check.

Judges. Route, Objects, Structure, Content and Appearance are scored by MLLM judges. Specifically, we use two of the strongest available MLLMs, Claude Opus 5.5 (xhigh effort) and Claude Fable 5.2 (medium effort), as the two judges; both differ from the MLLM used by our agent (GPT-6-Astra). Geometric failures are computed directly from the generated geometry.

For each metric, the judges receive only the input image, video frames sampled at 2 fps, and the corresponding evaluation question in a separate call. For Route and Objects, the question includes the fixed checklist derived from the instruction. Scores are averaged over the two judges and then over instructions. The two judges give the same answer on 94% of route steps and 92% of objects, and 97% of their consistency ratings differ by at most one point.

We add the detailed metrics and the prompt for judges in the appendix.

2. Results

Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑) and geometric failures (% of chunks ↓)

Route ↑Objects ↑Structure ↑Content ↑Appearance ↑Geometric failures ↓
Full system1.000.933.403.053.200%
w/o agent0.880.792.552.302.4521%
w/o world state0.960.933.382.832.8811%
w/o state update0.960.863.002.953.008%
w/o geometric grounding0.990.883.353.102.9514%

The full system is first in every column except content consistency, where it is second. Compared with the full system:

  • w/o agent plans the whole path once and uses the instruction as every prompt. It is the worst on every metric, by a wide margin: route completion drops to 0.88, objects to 0.79, consistency by 0.75 to 0.85, and geometric failures rise to 21%.
  • w/o world state plans each chunk only from what has been generated so far. Its route and object completion stay close to the full system, but its content and appearance consistency fall to the lowest after w/o agent (3.05 → 2.83, 3.20 → 2.88), and geometric failures rise to 11%.
  • w/o state update generates the whole video from the initial plan, without review. Structure consistency drops (3.40 → 3.00), and geometric failures rise to 8%.
  • w/o geometric grounding uses neither the point cloud nor the geometric tools. Geometric failures rise from 0% to 14%.

Trigger frequencies (per chunk)

RegenerationLayout revisionRoute replanning
Full system0.060.120.77
w/o geometric grounding0.080.060.62

W/o agent, w/o world state and w/o state update have none of these steps.

AuthorsOpenReview ↗

W1: Related work; differences from WonderJourney, WonderVerse, and LingBot-World 2.0

We thank the reviewer for highlighting these related works. These systems differ from AgiWorld in their task objectives, agent–generator interfaces and agent-generator co-design.

1. Comparison with WonderJourney

WonderJourney combines LLM-generated scene descriptions with image outpainting and point-cloud rendering, using a VLM to validate the generated scenes. AgiWorld generates camera-controlled video directly, with the agent maintaining a world state and planning a coherent journey through the generated environment. Since its agent is designed to propose successive scenes, evaluating it on our route-and-object completion task would require adapting the agent's planning objective.

2. Comparison with WonderVerse

WonderVerse is not open-source. Its LLM guides scene extension with existing video generators, whose outputs are reconstructed into a 3D scene. AgiWorld-Agent autonomously plans and completes an instructed journey, maintains observed and imagined content in an explicit world state, and revises its plans from generated observations and geometry. We co-design this agent with a new camera-controlled video world model, AgiWorld-WM, so that planning, generation, and spatial memory continually inform one another.

3. Comparison with LingBot-World 2.0

LingBot-World 2.0 has not open-sourced its agent code. Its agent supplies action and event signals through the video model's existing control interface to drive interactive simulation. We co-design AgiWorld-Agent with our new camera-controlled video world model, AgiWorld-WM, around structured prompts, metric camera trajectories, and persistent spatial memory.

We assess AgiWorld-Agent's contribution through controlled ablations with the same world model (our response to Q1). At the world-model level, we additionally evaluate VMem on all 94 scenes under the Table 1 protocol, where AgiWorld-WM performs better on every metric on both benchmarks (our response to Q4).

AuthorsOpenReview ↗

W2 / Q2: Computational overhead and MLLM backbone sensitivity

1. Computational overhead

AgiWorld-WM runs on a single node with eight A100 GPUs. AgiWorld-Agent calls GPT-6-Astra through the Azure OpenAI Responses API, so the reported wall-clock time includes API response time.

Average overhead per 60-s rollout

Wall-clockCallsTokens (in / out)API cost
AgiWorld-Agent2 h 03 min75 LLM calls1.34M / 0.27M$23.6
AgiWorld-WM2 h 16 min12.8 chunk generations, 40 denoising steps each––
Total4 h 19 min$23.6

The 12.8 chunk generations include regeneration attempts for the 12 output chunks. The reported API cost covers MLLM usage and excludes GPU compute costs.

2. MLLM backbone sensitivity

We replace the MLLM in AgiWorld-Agent while keeping AgiWorld-WM fixed and use the instruction-following evaluation protocol described in our response to Q1.

Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑), ** geometric failures** (% of chunks ↓), and token consumption and API cost per 60-s rollout

Agent MLLMRoute ↑Objects ↑Structure ↑Content ↑Appearance ↑Geometric failures ↓Tokens (in / out)API cost
GPT-6-Astra1.000.933.403.053.200%1.34M / 0.27M$23.6
Claude Sonnet 5.50.900.843.353.153.006%2.30M / 0.29M$6.7
Qwen3.8-27B (open-weight)0.900.833.103.052.8510%1.60M / 0.57M–
Qwen3-VL-235B-A22B (open-weight)0.740.672.172.172.4224%1.44M / 0.12M–

GPT-6-Astra performs best overall. Claude Sonnet 5.5 and the open-weight Qwen3.8-27B also achieve comparable performance, with lower instruction-completion scores and higher geometric-failure rates. Claude Sonnet 5.5 incurs less than one third of GPT-6-Astra's API cost.

The revision adds both tables and the evaluation details to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the closed-loop co-design of planning and camera-controlled video generation (7JVj, FoT9), the memory mechanisms addressing occlusion and accumulated geometric error (7JVj, 9D4p, FoT9), and the strong empirical performance of AgiWorld-WM (9D4p, FoT9).

The main shared concern was limited quantitative evaluation for the complete agentic system. We have added an evaluation of the full system and four agent ablations on 20 image–instruction pairs, reporting instruction completion, world consistency, and geometric failures in the final outputs (7JVj Q1; 9D4p Q1; FoT9 Q1). We also report regeneration frequencies and generation budgets (9D4p Q1), and compare the closed-loop system against an open-loop planner with resampling under the same regeneration limits (FoT9 Q2).

Two reviewers also requested a direct evaluation of revisit consistency. We now evaluate AgiWorld-WM’s revisit consistency by comparing generated views from earlier and delayed visits under the common prescribed trajectories used in Table 1. Results are grouped by elapsed time (9D4p Q2; FoT9 Q3) and camera travel distance (FoT9 Q3).

We further report computational overhead and MLLM backbone sensitivity (7JVj Q2), re-evaluate geometric metrics with VGGT and MASt3R and clarify the use of requested poses (7JVj Q3), and add a quantitative comparison with VMem (7JVj Q4). The related-work discussions are expanded to clarify differences from prior agentic world models (7JVj W1) and memory-based methods (7JVj Q4).

All new results, evaluation protocols, and related-work additions will be added into the revision. Detailed responses follow under each review.

AuthorsOpenReview ↗

Revised paper uploaded

We have uploaded the revised paper, with all changes marked in blue. Besides the content of our responses, the revision adds the full judge prompts used in the quantitative evaluation of the agentic system (Appendix E.5), and Figure 11, which directly shows revisit examples on DL3DV-Evaluation (Appendix D.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.