- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces the World Turing Machine (WTM), a persistent read-write memory for pose-conditioned spatial world modeling. WTM represents each scene using fixed-capacity Address-Content slots. Before incorporating a new observation, a Reader predicts its DINOv2 patch features from the current memory and camera pose. A Writer then reuses the Reader's routing weights to write the prediction residual into Content and pose/depth-derived geometric evidence into Address. Local neighboring-view prediction and global replay/unvisited-view objectives train the memory updates without direct supervision of the memory state. At inference time, the Reader supports both forward feature prediction at queried poses and inverse viewpoint matching through candidate-pose search. Experiments cover held-out-view prediction on ScanNet, cross-dataset transfer, ViewSuite viewpoint tasks, internal design ablations, and a synthetic pose-time extension.
Critical strengths
-
A coherent predictive read-write mechanism for persistent scene memory. The Reader does more than retrieve stored observations: it predicts the incoming view and produces routing weights that the Writer reuses for residual correction. Separating Address from Content allows the model to adapt both how evidence is accessed and what evidence is retained. This is a technically meaningful design, and the training-objective ablations in Table 2 support the complementary roles of local acquisition, replay, and unvisited-view supervision.
-
Broad validation of a shared forward/inverse interface. The same Reader is used for held-out-view feature prediction, pose-to-view matching, view-to-pose matching, and iterative localization. The paper also includes cross-dataset evaluation, memory/routing ablations, retention analysis, teacher-feature ablations, and comparisons with latent prediction methods such as Spa3R and RAE-NWM. This breadth makes the work more consequential than a single-benchmark architectural result.
Critical weaknesses
-
The evaluation does not isolate the learned memory from a simpler geometry-aligned feature memory. WTM receives ground-truth camera poses and pseudo-depth, and the Writer explicitly encodes reference-frame 3D patch coordinates when updating Address. A direct DINO-feature point cloud, voxel map, or geometry-indexed token bank using the same observations, poses, depth, memory budget, and query protocol is therefore a crucial missing baseline. The RGB reconstruction baselines in Table 1 have different output interfaces, and evaluating their outputs after conversion to WTM's native DINOv2 space does not answer whether learned Address-Content updates outperform direct geometric feature accumulation. The matched Spa3R and RAE-NWM comparisons are useful, but neither provides this simple control. This omission also weakens the claim that spatial organization "emerges," since explicit 3D evidence is injected into Address.
-
The ViewSuite results conflate representation quality with a high-budget, task-designed search procedure. The IVP system uses validation-tuned thresholds, deterministic frontier exploration, macro actions, valid local action sequences, and approximately 2,250 candidate-view queries per movement step for the main setting. Replacing WTM with VGGT/FSM under the same search rules is a useful control, but it does not make the comparison with ViewAgent compute-matched. The headline IVP improvement over ViewAgent is only 48.5% versus 47.8%, with no uncertainty estimate, while the reported 13.1-point average gain is driven mainly by the one-shot P2V/V2P tasks. In addition, the stated 120M parameter count excludes the frozen DINOv2 encoder and depth estimator. Without query-matched compute, latency, full model cost, and statistical uncertainty, the evidence for superior spatial decision making is not yet convincing.
Questions for authors
-
Additional evidence that could materially change my assessment: Can the authors compare WTM with a direct geometry-aligned DINO feature memory, such as feature back-projection into a point cloud or voxel map, using exactly the same four observations, ground-truth poses, pseudo-depth, memory budget, and query/evaluation protocol? A similarly matched geometry-indexed token-bank baseline would also be informative.
-
Clarification and additional evidence: What environment information is used to enumerate valid local action sequences and global candidate positions in ViewSuite? Please report total candidate queries, wall-clock latency or FLOPs, and the cost of the frozen DINOv2/depth modules. How does WTM compare with ViewAgent or other baselines under a matched inference budget, and is the 48.5% versus 47.8% IVP difference statistically significant across episodes or seeds?
-
Clarification: In what sense is the spatial organization of Address emergent when the Address updater receives pose/depth-derived reference-frame 3D coordinates? Can the authors report the routing-correlation and cross-view-correspondence analyses when depth or explicit 3D coordinate evidence is removed?
6 comments
Q2 & W2. IVP Performance and Computational Cost
We thank the reviewer for raising concerns about the planner and its computational cost. We revisited our implementation, simplified the planner, and added inference time and TFLOPs to Table 3.
Table 3. Performance on ViewSuite. Time and TFLOPs are reported per IVP decision.
| Method | Parameters | P2V | V2P | IVP | All | Time (s) | TFLOPs |
|---|---|---|---|---|---|---|---|
| General VLMs | |||||||
| GPT-5.4 Pro | -- | 53.1 | 50.7 | 18.5 | 40.8 | -- | -- |
| Qwen3.5 | 397B | 39.7 | 35.5 | 4.3 | 26.5 | -- | -- |
| Qwen2.5-VL | 7B | 29.5 | 24.2 | 2.5 | 18.7 | -- | -- |
| Task-specifically Trained VLM | |||||||
| ViewAgent | 7B | 57.3 | 52.8 | 47.8 | 52.6 | 1.2 | 68.9 |
| Iterative Inverse Matching | |||||||
| VGGT-Ω | 1.1B | -- | -- | 19.1 | -- | 15.6 | 291.5 |
| FSM-LVSM-3D | 0.31B | -- | -- | 16.2 | -- | 69.9 | 1985 |
| WTM-pose (Ours) | 0.12B | 66.4 | 82.3 | 56.8 | 68.5 | 2.3 | 77.0 |
We address the concerns from three aspects:
1. Planner design. The reviewer concerns that our planner is "task-designed". However, our planner was intentionally designed as a simple interface to apply view-prediction models to IVP, without training a task-specific policy. This also allows us to compare WTM, VGGT-Ω, and FSM-LVSM-3D under the same search procedure. In the updated submission, we further simplify the planner by removing the original global search stage and keeping only greedy local search. At each step, the planner searches nearby candidate views with a short horizon of (H=3) and executes an action toward the best match. We consider the planner an evaluation interface, rather than a contribution of this work.
2. Performance and robustness. Interestingly, the simpler planner also improves WTM's IVP success rate to 56.8%, about 9 percentage points higher than the task-trained ViewAgent (47.8%). Regarding statistical uncertainty, both WTM and our greedy planner are deterministic at inference time. We then independently train WTM with three different random seeds to evaluate the stability across seeds.
| Method | IVP Success Rate (%) ↑ |
|---|---|
| ViewAgent | 47.80 |
| WTM (seed 7, original) | 56.79 |
| WTM (seed 42) | 58.49 |
| WTM (seed 52) | 57.92 |
| WTM (Mean ± Std) | 57.73 ± 0.87 |
WTM consistently outperforms ViewAgent across all three training seeds, achieving an average advantage of +9.93 percentage points, with a small standard deviation of 0.87 pp. This further supports the stability of our IVP performance improvement.
3. Computational cost. As for computational cost, WTM has the lowest running time and FLOPs among the three 3D models using the greedy planner. (3 times less than VGGT-)
Among the VLMs, ViewAgent is trained with ViewSuite data via SFT+RL (72 hours on 8xH100 GPUs), so that its capability on this task comes from the extra training, whereas the 3D models are not specially trained on this task. Hence we put ViewAgent here as a reference, rather than a fair comparison. Considering that, we still achieve much better IVP results (56.8 vs. 47.8), with cost comparable to ViewAgent's (77.0 vs. 68.9 TFLOPs), although ViewAgent achieves lower inference latency (1.2 s vs. 2.3 s) with optimized VLM inference backends. WTM is also substantially more efficient than VGGT-Ω and FSM-LVSM-3D under the same search procedure.
We also measured the component-wise cost of WTM. DA3 and DINOv2 account for only 1.9 TFLOPs combined, while WTM's candidate-view predictions account for 74.1 TFLOPs out of the total 77.0 TFLOPs per decision.
The peak GPU memory usage during IVP inference: 21.86 GiB for WTM and 66.67 GiB for ViewAgent (SGLang). Thus, WTM requires approximately 3× less peak GPU memory than ViewAgent in our evaluated configurations.
We also provide the requested breakdown of WTM's per-decision computational cost:
| Component | Parameters (M) | Time (s) | TFLOPs |
|---|---|---|---|
| DA3 | 334 | 0.04 | 1.2 |
| DINOv2 | 304 | 0.03 | 0.7 |
| WTM | 120 | 1.7 | 74.1 |
| Others (candidate generation, matching) | -- | 0.5 | 0.9 |
| Total | 758 | 2.3 | 77.0 |
The total accounts for overlapping operations. As shown, the costs of DA3 and DINOv2 are relatively small, with most computation spent on WTM's candidate-view predictions.
Q2(environmental information) & Q3
Q2. Environment Information Used During IVP
Thanks for raising this point. We agree that the environment feedback used by the planner should be made clearer. We have added the following clarification to Appendix A.4:
Candidate poses are generated only from the current camera pose and the predefined action space, without accessing ground-truth scene geometry or occupancy information. Apart from the new RGB observation returned after each action, the only additional environment feedback used by the planner is a failure signal when the agent attempts to move beyond the scene boundary, upon which the planner selects an alternative action. This procedure is applied consistently to all evaluated methods.
Q3. Emergent Spatial Organization
Q3.1 Explanation of the term "Emergent"
We use the term "emergent spatial organization" for two main reasons.
1. The spatial organization is learned, not explicitly assigned. We do not directly supervise the memory slots or assign any predefined 3D coordinates to them. Instead, through end-to-end training, different slots learn to specialize in different spatial regions. As shown in Table 4 and Figure 5, the learned routing mechanism tends to associate spatially corresponding patches with the same memory slots. We refer to this learned spatial division of responsibility among slots as emergent organization, since it arises without explicit slot-level spatial supervision.
2. The Reader learns to infer spatial regions without query-view depth. Although the observed images are provided with camera poses and pseudo-depth, the Reader receives only the camera pose for a queried view, without its depth. The camera pose determines the viewing rays, but not which surfaces lie along them. A ray alone cannot distinguish between the front and back surfaces of an object, or empty space along the ray. Yet WTM learns to use its memory to infer the relevant spatial regions and route queries to the corresponding slots.
Together, these observations suggest that WTM learns a spatially organized memory from view-prediction supervision, without explicitly imposing a 3D structure or coordinates on its memory slots.
Q3.2 Spatial Organization Without Depth Information
To address your concern about the spatial routing when depth is removed. We trained an additional WTM variant without depth information (WTM w/o Depth). Importantly, this model is trained from scratch with the depth branch completely removed from the Address Writer, rather than simply disabling depth during evaluation. We repeated the same routing correlation analysis (Table 4) and cross-view correspondence evaluation (Table 15) for this model.
1. Routing correlation (Spearman's ρ)(Corresponding to Tab4)
| Model | 3D Location | Viewing Ray | Visual Feature | Image Grid |
|---|---|---|---|---|
| Original WTM | 0.3072 | 0.2640 | 0.1942 | 0.1680 |
| WTM w/o Depth | 0.2989 | 0.2631 | 0.1705 | 0.1462 |
2. Cross-view correspondence (AUROC, τ = 0.20 m)(Corresponding to Tab15)
| Model | Grid | DINO | Address Score | Content Score | Joint Routing |
|---|---|---|---|---|---|
| Original WTM | 0.4895 | 0.8562 | 0.9445 | 0.8856 | 0.9469 |
| WTM w/o Depth | 0.4895 | 0.8562 | 0.9302 | 0.8711 | 0.9332 |
Without any depth input, WTM still develops a similar spatial organization. The routing correlation with 3D location remains the highest among the four attributes (0.299 vs. 0.307 in the original model), and the joint routing still achieves 0.933 AUROC on cross-view correspondence, compared with 0.947 for the original WTM and 0.856 for DINO features alone.
The task performance of the no-depth variant is already reported in Appendix Table 9:
| Method | Cosine ↑ | R@1 (%) ↑ |
|---|---|---|
| Full WTM | 0.662 | 86.07 |
| WTM w/o Depth | 0.651 | 77.42 |
Removing depth reduces task performance, which is expected since pseudo-depth provides additional geometric information. However, despite this performance drop, the learned spatial organization remains largely preserved, as demonstrated by the routing correlation and cross-view correspondence results above.
Together, these results suggest that pseudo-depth helps strengthen the learned spatial organization, but is not necessary for its emergence. This further supports our use of the term "emergent": memory slots are not assigned predefined 3D coordinates or directly supervised to represent particular spatial regions, but develop such organization through training.
Q1 & W1. Comparison with Simple Feature Back-projection Baselines
We agree with the reviewer that this is a meaningful baseline to help isolate the contribution of our memory mechanism from simply storing geometry-aligned features. We implemented two variants of a direct geometry-aligned DINO feature memory:
All baselines use the same four observed images, ground-truth camera poses, pseudo-depth, and DINOv2 features as WTM, and follow the same query and retrieval evaluation protocol. At query time, we project the stored 3D features into the target view, retaining the frontmost feature at each projected patch. Uncovered patches are handled using nearest point to the viewing ray. We evaluate both full memory (storing all observed patches) and budget-matched variants compressed to the same 3 MiB memory budget as WTM.
WTM clearly outperforms both budget-matched baselines. Even when the point-cloud baseline stores all observed patches using over five times more memory (16.06 MiB), it still performs worse than WTM. We thank the reviewer for suggesting this useful comparison. These results support that WTM's advantage comes from its learned memory organization and read/write mechanism, rather than simply storing visual features in a geometry-aligned memory.