- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AgiWorld, a system that builds and explores a persistent world from a single image and a free-form instruction. It combines AgiWorld-Agent, which maintains a high-level world state and plans the journey, with AgiWorld-WM, an autoregressive video model equipped with spatial memory. The world model introduces occlusion-aware history selection and reliability-based memory maintenance. Experiments on DL3DV-Evaluation and the proposed World Explore benchmark show good video quality and camera-following results, while qualitative examples illustrate the complete system.
Critical strengths
-
The division between a semantic world state maintained by the agent and a geometric memory maintained by the video model is sensible. The two components have a clear interface through metric camera trajectories, prompts, and generated geometry.
-
The world-model component is technically strong. AgiWorld-WM performs well across visual quality, camera following, and reprojection metrics, and the proposed memory mechanisms address concrete long-rollout problems: occlusion and accumulated geometric error.
Critical weaknesses
-
The complete agentic system is evaluated mainly through a few qualitative examples. Most quantitative experiments evaluate AgiWorld-WM under given trajectories and prompts, rather than the central claim that the closed-loop agent can reliably convert high-level instructions into persistent worlds. The main evaluation contains only a small number of complete-system examples and no aggregate instruction-completion or semantic-consistency results.
-
Although the evaluation includes revisiting trajectories, the metrics do not directly measure consistency between an initial visit and a later revisit. FID and short-interval depth reprojection are useful but do not establish that a place remains the same after a long absence. The system can also regenerate failed chunks, but the observed retry frequency and computational overhead are not reported, making it hard to separate state-update gains from additional sampling.
Questions for authors
-
Can the authors report aggregate full-system results over a reasonably sized instruction set, including completion rate, realization of requested objects, long-range revisitation consistency, and failure rate, including quantitative evaluation of the existing agent ablations?
-
How often does chunk regeneration occur? Under an equal generation budget, how does the closed-loop system compare with an open-loop planner plus simple resampling?
-
Can the authors directly compare a region at its initial visit and a later revisit, grouped by the elapsed time or camera travel distance between visits?
5 comments
Q2: Regeneration overhead and comparison with open-loop resampling
We added the requested open-loop + resampling comparison on the same 20 image–instruction pairs used in our response to Q1.
The baseline keeps its initial plan fixed, as in w/o state update, and performs no MLLM review or state updates. When a chunk fails the same camera checks used by the full system (Appendix A.6), it regenerates the chunk using the same camera trajectory and prompt with a new seed. It uses the same regeneration limits as the full system.
The full system regenerates 6% of its chunks, corresponding to 1.06 world-model generations per output chunk. Open loop + resampling averages 1.03 under the same regeneration limits.
Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑), geometric failures (% of chunks ↓), world-model generations per chunk, and LLM tokens per 60-s rollout
| Route ↑ | Objects ↑ | Structure ↑ | Content ↑ | Appearance ↑ | Geometric failures ↓ | WM generation / chunk | Tokens (in / out) | |
|---|---|---|---|---|---|---|---|---|
| Full system | 1.00 | 0.93 | 3.40 | 3.05 | 3.20 | 0% | 1.06 | 1.34M / 0.27M |
| open loop (w/o state update) | 0.96 | 0.86 | 3.00 | 2.95 | 3.00 | 8% | 1.00 | 0.10M / 0.03M |
| open loop + resampling | 0.98 | 0.86 | 3.10 | 2.95 | 3.05 | 5% | 1.03 | 0.10M / 0.03M |
Adding resampling reduces geometric failures from 8% to 5%, but leaves the Objects and Content scores unchanged. The full system achieves an Objects score of 0.93, higher consistency scores across all three dimensions, and a geometric-failure rate of 0%, using 1.06 world-model generations per output chunk compared with 1.03 for open loop + resampling.
The full system uses more LLM tokens to review generated observations, update its world state, and replan using the generated geometry. We report this additional agent cost explicitly. Under the same regeneration limits, these results support the benefit of closed-loop review and replanning over resampling a fixed initial plan.
W2 / Q3: Revisit consistency grouped by elapsed time and camera travel distance
We agree that FID and short-interval depth reprojection do not show that a place stays the same after a long absence. We therefore added an evaluation that directly compares two generated views of the same place: one from the earlier visit and one from the delayed revisit. This evaluates whether previously generated scene content is preserved when the camera returns after a delay.
We select revisit pairs on DL3DV-Evaluation, whose trajectories return to where they started. A frame at 15 s or later forms a revisit pair with an earlier frame, generated 15 to 60 seconds before, if their requested cameras are within 1.5 m in position and 60° in viewing direction. If several earlier frames qualify, we choose the one with the smallest camera-position distance. Frames without a qualifying earlier view are excluded. We warp the earlier frame into the revisit view and compute LPIPS between the two generated frames on the non-sky pixels visible in both. All methods are scored on the same pairs, using their Table 1 rollouts. For the distance-based analysis, we group the same revisit pairs by the cumulative distance traveled along the camera trajectory between the two visits.
Revisit consistency (LPIPS ↓, grouped by elapsed time)
| 15–30 s | 30–45 s | 45–60 s | All | |
|---|---|---|---|---|
| SPMem | .421 | .449 | .458 | .440 |
| GEN3C | .326 | .348 | .318 | .331 |
| HY-WorldPlay | .520 | .586 | .607 | .566 |
| LingBot-World | .500 | .537 | .539 | .523 |
| Matrix-Game 3.0 | .450 | .468 | .466 | .460 |
| Lyra 2.0 | .277 | .320 | .316 | .302 |
| SANA-WM | .321 | .340 | .344 | .333 |
| AgiWorld-WM (ours) | .277 | .298 | .296 | .289 |
Revisit consistency (LPIPS ↓, grouped by camera travel)
| LPIPS ↓ | 5–15 m | 15–25 m | ≥25 m |
|---|---|---|---|
| SPMem | .436 | .457 | .409 |
| GEN3C | .328 | .342 | .312 |
| HY-WorldPlay | .544 | .590 | .577 |
| LingBot-World | .519 | .538 | .495 |
| Matrix-Game 3.0 | .467 | .466 | .426 |
| Lyra 2.0 | .276 | .321 | .328 |
| SANA-WM | .328 | .341 | .325 |
| AgiWorld-WM (ours) | .275 | .302 | .282 |
AgiWorld-WM matches Lyra 2.0 at 15–30 s and achieves the lowest revisit error at 30–45 s, 45–60 s, and overall. These results provide direct evidence that our method better preserves previously generated content across longer revisit delays. Grouped by camera travel, AgiWorld-WM has the lowest error in every range, and its margin over the strongest baseline grows with the distance traveled: .001 at 5–15 m, .019 at 15–25 m and .030 at ≥25 m.
Q1: Quantitative Evaluation of AgiWorld-Agent
1. Evaluation protocol
To evaluate the complete agentic system, we construct an instruction-following test set and an evaluation protocol that measures instruction completion, world consistency and geometric failures. The test set has 20 image–instruction pairs of indoor and outdoor scenes. Each instruction asks for a route and names objects that should appear along it, e.g., "walk out through the open garage door and down the driveway, passing the silver car on your left." We evaluate the full system and the four agent ablations of Figure 9. All variants use the same input image, instruction, world model, and initial seed. Each ablation is capped at the number of output chunks produced by the full system.
Metrics.
Judges. Route, Objects, Structure, Content and Appearance are scored by MLLM judges. Specifically, we use two of the strongest available MLLMs, Claude Opus 5.5 (xhigh effort) and Claude Fable 5.2 (medium effort), as the two judges; both differ from the MLLM used by our agent (GPT-6-Astra). Geometric failures are computed directly from the generated geometry.
For each metric, the judges receive only the input image, video frames sampled at 2 fps, and the corresponding evaluation question in a separate call. For Route and Objects, the question includes the fixed checklist derived from the instruction. Scores are averaged over the two judges and then over instructions. The two judges give the same answer on 94% of route steps and 92% of objects, and 97% of their consistency ratings differ by at most one point.
2. Results
Instruction completion (checklist score, 0–1 ↑), world consistency (1–5 ↑) and geometric failures (% of chunks ↓)
The full system is first in every column except content consistency, where it is second. Compared with the full system:
The evaluation above measures instruction completion and world consistency under agent-generated journeys. Separately, our response to Q3 evaluates long-range revisit consistency of AgiWorld-WM under common prescribed trajectories, grouping revisit pairs by elapsed time and camera travel distance.