acceptodds
Withdrawn from ICLR 2027

World Turing Machine: Learning Read–Write Memory for 3D Worlds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

Spatial world modeling requires collecting evidence across viewpoints and maintaining it in a persistent state for predictions at unobserved views. However, organizing and accessing such accumulated information over time remains a central challenge. In this study, we introduce the World Turing Machine (WTM), which addresses this through learned read–write computation over an Address–Content memory. This memory evolves with observations while the learned read–write operations remain fixed at inference. Before each write, the Reader predicts incoming visual features from the current memory and the condition. The Writer reuses the Reader's access weights to correct Content with prediction residuals and update Address with observation-derived geometric evidence. Local and global predictive objectives train acquisition, retention, and inference across viewpoints without directly supervising memory. The same Reader supports forward prediction at unvisited poses and inverse pose matching against target-frame features, without an additional localization head. On ScanNet, WTM outperforms evaluated novel-view synthesis and feed-forward 3D reconstruction baselines in held-out feature prediction. With a simple search over candidate poses, WTM achieves 65.7% success on ViewSuite tasks without task-specific policy training, surpassing ViewAgent by +13.1 percentage points. Analyses reveal emergent spatial organization during online memory formation and show that the learned read–write mechanism generalizes across 3D environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
View Planning, Neural Turing Machine, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5rMrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the World Turing Machine (WTM), a persistent read-write memory for pose-conditioned spatial world modeling. WTM represents each scene using fixed-capacity Address-Content slots. Before incorporating a new observation, a Reader predicts its DINOv2 patch features from the current memory and camera pose. A Writer then reuses the Reader's routing weights to write the prediction residual into Content and pose/depth-derived geometric evidence into Address. Local neighboring-view prediction and global replay/unvisited-view objectives train the memory updates without direct supervision of the memory state. At inference time, the Reader supports both forward feature prediction at queried poses and inverse viewpoint matching through candidate-pose search. Experiments cover held-out-view prediction on ScanNet, cross-dataset transfer, ViewSuite viewpoint tasks, internal design ablations, and a synthetic pose-time extension.

Critical strengths
  1. A coherent predictive read-write mechanism for persistent scene memory. The Reader does more than retrieve stored observations: it predicts the incoming view and produces routing weights that the Writer reuses for residual correction. Separating Address from Content allows the model to adapt both how evidence is accessed and what evidence is retained. This is a technically meaningful design, and the training-objective ablations in Table 2 support the complementary roles of local acquisition, replay, and unvisited-view supervision.

  2. Broad validation of a shared forward/inverse interface. The same Reader is used for held-out-view feature prediction, pose-to-view matching, view-to-pose matching, and iterative localization. The paper also includes cross-dataset evaluation, memory/routing ablations, retention analysis, teacher-feature ablations, and comparisons with latent prediction methods such as Spa3R and RAE-NWM. This breadth makes the work more consequential than a single-benchmark architectural result.

Critical weaknesses
  1. The evaluation does not isolate the learned memory from a simpler geometry-aligned feature memory. WTM receives ground-truth camera poses and pseudo-depth, and the Writer explicitly encodes reference-frame 3D patch coordinates when updating Address. A direct DINO-feature point cloud, voxel map, or geometry-indexed token bank using the same observations, poses, depth, memory budget, and query protocol is therefore a crucial missing baseline. The RGB reconstruction baselines in Table 1 have different output interfaces, and evaluating their outputs after conversion to WTM's native DINOv2 space does not answer whether learned Address-Content updates outperform direct geometric feature accumulation. The matched Spa3R and RAE-NWM comparisons are useful, but neither provides this simple control. This omission also weakens the claim that spatial organization "emerges," since explicit 3D evidence is injected into Address.

  2. The ViewSuite results conflate representation quality with a high-budget, task-designed search procedure. The IVP system uses validation-tuned thresholds, deterministic frontier exploration, macro actions, valid local action sequences, and approximately 2,250 candidate-view queries per movement step for the main H=4H=4 setting. Replacing WTM with VGGT/FSM under the same search rules is a useful control, but it does not make the comparison with ViewAgent compute-matched. The headline IVP improvement over ViewAgent is only 48.5% versus 47.8%, with no uncertainty estimate, while the reported 13.1-point average gain is driven mainly by the one-shot P2V/V2P tasks. In addition, the stated 120M parameter count excludes the frozen DINOv2 encoder and depth estimator. Without query-matched compute, latency, full model cost, and statistical uncertainty, the evidence for superior spatial decision making is not yet convincing.

Questions for authors
  1. Additional evidence that could materially change my assessment: Can the authors compare WTM with a direct geometry-aligned DINO feature memory, such as feature back-projection into a point cloud or voxel map, using exactly the same four observations, ground-truth poses, pseudo-depth, memory budget, and query/evaluation protocol? A similarly matched geometry-indexed token-bank baseline would also be informative.

  2. Clarification and additional evidence: What environment information is used to enumerate valid local action sequences and global candidate positions in ViewSuite? Please report total candidate queries, wall-clock latency or FLOPs, and the cost of the frozen DINOv2/depth modules. How does WTM compare with ViewAgent or other baselines under a matched inference budget, and is the 48.5% versus 47.8% IVP difference statistically significant across episodes or seeds?

  3. Clarification: In what sense is the spatial organization of Address emergent when the Address updater receives pose/depth-derived reference-frame 3D coordinates? Can the authors report the routing-correlation and cross-view-correspondence analyses when depth or explicit 3D coordinate evidence is removed?

6 comments
AuthorsOpenReview ↗

Q1 & W1. Comparison with Simple Feature Back-projection Baselines

We agree with the reviewer that this is a meaningful baseline to help isolate the contribution of our memory mechanism from simply storing geometry-aligned features. We implemented two variants of a direct geometry-aligned DINO feature memory:

  • Point: Each observed image patch is back-projected into 3D as a point carrying its DINO feature.
  • Voxel: The projected points are further merged into voxels of size 0.20 m.

All baselines use the same four observed images, ground-truth camera poses, pseudo-depth, and DINOv2 features as WTM, and follow the same query and retrieval evaluation protocol. At query time, we project the stored 3D features into the target view, retaining the frontmost feature at each projected patch. Uncovered patches are handled using nearest point to the viewing ray. We evaluate both full memory (storing all observed patches) and budget-matched variants compressed to the same 3 MiB memory budget as WTM.

MethodMemoryCosine ↑R@1 ↑
WTM3 MiB0.52826.91
3D Feature Point Cloud16.06 MiB0.31722.38
3D Feature Point Cloud (budget-matched)3 MiB0.32711.76
3D Voxel Feature Memory (budget-matched)3 MiB0.32911.48

WTM clearly outperforms both budget-matched baselines. Even when the point-cloud baseline stores all observed patches using over five times more memory (16.06 MiB), it still performs worse than WTM. We thank the reviewer for suggesting this useful comparison. These results support that WTM's advantage comes from its learned memory organization and read/write mechanism, rather than simply storing visual features in a geometry-aligned memory.

AuthorsOpenReview ↗

Q2 & W2. IVP Performance and Computational Cost

We thank the reviewer for raising concerns about the planner and its computational cost. We revisited our implementation, simplified the planner, and added inference time and TFLOPs to Table 3.

Table 3. Performance on ViewSuite. Time and TFLOPs are reported per IVP decision.

MethodParametersP2VV2PIVPAllTime (s)TFLOPs
General VLMs
GPT-5.4 Pro--53.150.718.540.8----
Qwen3.5397B39.735.54.326.5----
Qwen2.5-VL7B29.524.22.518.7----
Task-specifically Trained VLM
ViewAgent7B57.352.847.852.61.268.9
Iterative Inverse Matching
VGGT-Ω1.1B----19.1--15.6291.5
FSM-LVSM-3D0.31B----16.2--69.91985
WTM-pose (Ours)0.12B66.482.356.868.52.377.0

We address the concerns from three aspects:

1. Planner design. The reviewer concerns that our planner is "task-designed". However, our planner was intentionally designed as a simple interface to apply view-prediction models to IVP, without training a task-specific policy. This also allows us to compare WTM, VGGT-Ω, and FSM-LVSM-3D under the same search procedure. In the updated submission, we further simplify the planner by removing the original global search stage and keeping only greedy local search. At each step, the planner searches nearby candidate views with a short horizon of (H=3) and executes an action toward the best match. We consider the planner an evaluation interface, rather than a contribution of this work.

2. Performance and robustness. Interestingly, the simpler planner also improves WTM's IVP success rate to 56.8%, about 9 percentage points higher than the task-trained ViewAgent (47.8%). Regarding statistical uncertainty, both WTM and our greedy planner are deterministic at inference time. We then independently train WTM with three different random seeds to evaluate the stability across seeds.

MethodIVP Success Rate (%) ↑
ViewAgent47.80
WTM (seed 7, original)56.79
WTM (seed 42)58.49
WTM (seed 52)57.92
WTM (Mean ± Std)57.73 ± 0.87

WTM consistently outperforms ViewAgent across all three training seeds, achieving an average advantage of +9.93 percentage points, with a small standard deviation of 0.87 pp. This further supports the stability of our IVP performance improvement.

3. Computational cost. As for computational cost, WTM has the lowest running time and FLOPs among the three 3D models using the greedy planner. (3 times less than VGGT-Ω\Omega)

Among the VLMs, ViewAgent is trained with ViewSuite data via SFT+RL (72 hours on 8xH100 GPUs), so that its capability on this task comes from the extra training, whereas the 3D models are not specially trained on this task. Hence we put ViewAgent here as a reference, rather than a fair comparison. Considering that, we still achieve much better IVP results (56.8 vs. 47.8), with cost comparable to ViewAgent's (77.0 vs. 68.9 TFLOPs), although ViewAgent achieves lower inference latency (1.2 s vs. 2.3 s) with optimized VLM inference backends. WTM is also substantially more efficient than VGGT-Ω and FSM-LVSM-3D under the same search procedure.

We also measured the component-wise cost of WTM. DA3 and DINOv2 account for only 1.9 TFLOPs combined, while WTM's candidate-view predictions account for 74.1 TFLOPs out of the total 77.0 TFLOPs per decision.

The peak GPU memory usage during IVP inference: 21.86 GiB for WTM and 66.67 GiB for ViewAgent (SGLang). Thus, WTM requires approximately 3× less peak GPU memory than ViewAgent in our evaluated configurations.

We also provide the requested breakdown of WTM's per-decision computational cost:

ComponentParameters (M)Time (s)TFLOPs
DA33340.041.2
DINOv23040.030.7
WTM1201.774.1
Others (candidate generation, matching)--0.50.9
Total7582.377.0

The total accounts for overlapping operations. As shown, the costs of DA3 and DINOv2 are relatively small, with most computation spent on WTM's candidate-view predictions.

AuthorsOpenReview ↗

Q2(environmental information) & Q3

Q2. Environment Information Used During IVP

Thanks for raising this point. We agree that the environment feedback used by the planner should be made clearer. We have added the following clarification to Appendix A.4:

Candidate poses are generated only from the current camera pose and the predefined action space, without accessing ground-truth scene geometry or occupancy information. Apart from the new RGB observation returned after each action, the only additional environment feedback used by the planner is a failure signal when the agent attempts to move beyond the scene boundary, upon which the planner selects an alternative action. This procedure is applied consistently to all evaluated methods.

Q3. Emergent Spatial Organization

Q3.1 Explanation of the term "Emergent"

We use the term "emergent spatial organization" for two main reasons.

1. The spatial organization is learned, not explicitly assigned. We do not directly supervise the memory slots or assign any predefined 3D coordinates to them. Instead, through end-to-end training, different slots learn to specialize in different spatial regions. As shown in Table 4 and Figure 5, the learned routing mechanism tends to associate spatially corresponding patches with the same memory slots. We refer to this learned spatial division of responsibility among slots as emergent organization, since it arises without explicit slot-level spatial supervision.

2. The Reader learns to infer spatial regions without query-view depth. Although the observed images are provided with camera poses and pseudo-depth, the Reader receives only the camera pose for a queried view, without its depth. The camera pose determines the viewing rays, but not which surfaces lie along them. A ray alone cannot distinguish between the front and back surfaces of an object, or empty space along the ray. Yet WTM learns to use its memory to infer the relevant spatial regions and route queries to the corresponding slots.

Together, these observations suggest that WTM learns a spatially organized memory from view-prediction supervision, without explicitly imposing a 3D structure or coordinates on its memory slots.

Q3.2 Spatial Organization Without Depth Information

To address your concern about the spatial routing when depth is removed. We trained an additional WTM variant without depth information (WTM w/o Depth). Importantly, this model is trained from scratch with the depth branch completely removed from the Address Writer, rather than simply disabling depth during evaluation. We repeated the same routing correlation analysis (Table 4) and cross-view correspondence evaluation (Table 15) for this model.

1. Routing correlation (Spearman's ρ)(Corresponding to Tab4)

Model3D LocationViewing RayVisual FeatureImage Grid
Original WTM0.30720.26400.19420.1680
WTM w/o Depth0.29890.26310.17050.1462

2. Cross-view correspondence (AUROC, τ = 0.20 m)(Corresponding to Tab15)

ModelGridDINOAddress ScoreContent ScoreJoint Routing
Original WTM0.48950.85620.94450.88560.9469
WTM w/o Depth0.48950.85620.93020.87110.9332

Without any depth input, WTM still develops a similar spatial organization. The routing correlation with 3D location remains the highest among the four attributes (0.299 vs. 0.307 in the original model), and the joint routing still achieves 0.933 AUROC on cross-view correspondence, compared with 0.947 for the original WTM and 0.856 for DINO features alone.

The task performance of the no-depth variant is already reported in Appendix Table 9:

MethodCosine ↑R@1 (%) ↑
Full WTM0.66286.07
WTM w/o Depth0.65177.42

Removing depth reduces task performance, which is expected since pseudo-depth provides additional geometric information. However, despite this performance drop, the learned spatial organization remains largely preserved, as demonstrated by the routing correlation and cross-view correspondence results above.

Together, these results suggest that pseudo-depth helps strengthen the learned spatial organization, but is not necessary for its emergence. This further supports our use of the term "emergent": memory slots are not assigned predefined 3D coordinates or directly supervised to represent particular spatial regions, but develop such organization through training.

Reviewer_3PPzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes World Turing Machine (WTM), a framework for learning persistent memory representations of 3D environments. WTM maintains an Address-Content memory through a Reader and a Writer. The Reader predicts visual features at queried viewpoints, while the Writer updates memory using prediction residuals and geometric evidence. Both modules share memory routing weights. The framework is trained using local and global predictive objectives without direct supervision of the memory state. Experiments on ScanNet evaluate feature prediction and viewpoint retrieval. The authors also apply WTM to ViewSuite for viewpoint matching and sequential view planning. Additional experiments investigate memory organization, cross-dataset generalization, and the effects of different training objectives.

Critical strengths

1.The proposed memory architecture is an interesting approach to 3D scene representation. The separation of Address and Content allows the model to update both memory organization and stored visual information as new observations arrive. The ablation results suggest that adaptive addressing plays an important role in viewpoint retrieval. 2.WTM supports both forward viewpoint prediction and inverse viewpoint matching without introducing a separate localization head. The evaluation on ScanNet and ViewSuite demonstrates the potential of using a shared memory representation for different spatial tasks.

Critical weaknesses

1.The shared routing mechanism between the Reader and Writer is a central component of WTM. However, the current experiments do not directly establish its advantage over independently learned read and write routing. Although the ablation studies demonstrate the importance of adaptive Address updates, they do not isolate the contribution of shared routing. This makes it difficult to determine how much of the performance improvement comes from the proposed coupling mechanism. 2.The performance improvements on some key evaluation metrics are relatively limited. In Table 1, WTM improves R@1 from 24.36% to 26.91% over VGGT-Ω. In Table 3, its IVP success rate is only 0.7 percentage points higher than ViewAgent. The paper does not provide statistical uncertainty estimates for these comparisons. Moreover, the reported IVP performance requires a considerable number of candidate-view queries. These limitations make it difficult to assess the significance and computational efficiency of the proposed approach.

Questions for authors

1.The R@1 values reported in Table 11 for DINOv2, DINOv3, V-JEPA 2.1, and CLIP are 86.07, 66.34, 52.23, and 44.38, respectively. These values exactly match the first four rows of Table 8, although the two tables evaluate different experimental factors. Could the authors verify these results and clarify whether the reported values are correct? 2.Could the authors provide a comparison between the proposed shared routing mechanism and independently learned write routing? This would help clarify the contribution of coupling memory reading and writing. 3.Table 1 shows that WTM substantially improves cosine similarity over VGGT-Ω (0.528 vs. 0.273), while the improvement in R@1 is relatively small (26.91% vs. 24.36%). Could the authors explain this discrepancy and further analyze retrieval performance at different distances between observed and queried viewpoints? This would help assess whether the improved feature predictions lead to better viewpoint discrimination in unobserved regions. 4.Table 13 reports approximately 2,250 candidate-view queries per movement step when the search horizon is four. Could the authors report the corresponding inference latency and GPU memory requirements? This information would help assess the computational cost of applying WTM to sequential view planning.

4 comments

Q1. Correction of Table 11

We mistakenly copied the R@1 values from another table when preparing Appendix Table 11. We have checked the original evaluation results and corrected the numbers in our updated submission. The correct R@1 values for DINOv2, DINOv3, V-JEPA 2.1, and CLIP are 86.07%, 80.95%, 83.83%, and 43.73%, respectively. The corresponding analysis was based on the correct results, so this correction does not affect our conclusions. We apologize for this mistake.

Q2 & W1. Independent Routing Ablation

We agree that the shared routing mechanism is an important part of WTM and deserves a more direct evaluation. Following the reviewer's suggestion, we implemented a variant with independently learned read and write routing. In this variant, the Writer's routing weights are independently predicted by a learned MLP conditioned on the query pose and current memory state, rather than reused from the pre-write Reader. All other configurations remain identical to the original WTM.

MethodCosine ↑R@1 ↑IVP with planner ↑
Shared read/write routing(Ours, Table 1)0.52826.91%56.8%
Independent read/write routing0.52723.21%49.1%

The independent-routing variant's Pose R@1 drops by 3.7 percentage points compared with the original WTM, and the its' IVP success rate with the same planner is only 49.1%. This result supports the benefit of coupling read and write routing, particularly for learning representations that preserve viewpoint discriminability.

For standard ablation variants comparison, in the paper we evaluate all variants using the WTM within-trajectory evaluation (Table 8). Here we also report the performance on thie setting, and we add this ablation into Appendix Table 8 in the updated submission:

MethodCosine ↑R@1 ↑
Shared routing(Ours)0.66286.07%
Independent routing0.60559.33%

The performance drop is even more pronounced in this setting, with a 26.74 pp decrease in R@1. Together, these results provide direct evidence for the benefit of shared read/write routing, particularly in learning memory representations that support accurate viewpoint retrieval and downstream planning.

We thank the reviewer for suggesting this experiment, which helps us better understand the contribution of our shared routing design.

AuthorsOpenReview ↗

Q3 & W2. View Prediction Performance and Uncertainty (Table 1)

We appreciate the reviewer's concerns about the difference between cosine similarity and retrieval improvements, as well as the statistical uncertainty. We address them with additional analysis below.

1. Why does cosine similarity improve much more than retrieval?

The two metrics measure different aspects of prediction quality. Cosine similarity measures how closely the predicted features align with the target features, whereas retrieval evaluates whether the target view ranks above competing views.

A prediction can become more similar to the target while also becoming more similar to other views in the same scene. Consequently, a substantial increase in target-feature cosine similarity need not produce a proportional improvement in retrieval accuracy: retrieval improves only when the relative ranking changes.

The two metrics capture different aspects of prediction quality, so improvements in cosine similarity need not translate proportionally into retrieval gains.

2. Statistical uncertainty across training seeds.

To assess the stability of our results, we trained WTM with two additional random seeds (42 and 52). We report the results of all three runs, together with their mean and standard deviation (Std).

MethodCosine Similarity ↑R@1 (%) ↑
VGGT-Ω (Table 1)0.27324.36
WTM (seed 7, Table 1)0.52826.91
WTM (seed 42)0.52927.73
WTM (seed 52)0.52827.54
WTM (Mean ± Std)0.5283 ± 0.000627.39 ± 0.43

The results are consistent across all three training runs, with a small standard deviation of only 0.43 percentage points in Pose R@1. On average, WTM outperforms VGGT-Ω by +0.255 in cosine similarity and +3.03 percentage points in Pose R@1. Interestingly, both additional seeds achieve higher Pose R@1 than our originally reported seed 7, and all three seeds outperform VGGT-Ω. This suggests that the improvement is stable across different training runs rather than being specific to the original random seed.

3. Performance at different viewpoint distances.

Following the reviewer's suggestion, we further group the test queries into four quartiles according to the distance between the observed and queried viewpoints.

Distance quartileWTM Cosine ↑VGGT-Ω Cosine ↑WTM R@1 (%) ↑VGGT-Ω R@1 (%) ↑
Q1 (nearest)0.5710.35346.6349.69
Q20.5290.28029.7024.25
Q30.5250.24922.1416.08
Q4 (farthest)0.4910.21811.5610.04
Overall0.5280.27326.9124.36

Note that the Overall row follows Table 1 and reports scene-macro averages, whereas the distance-binned results use query-level aggregation. Therefore, the average of the four quartiles does not necessarily match the Overall results.

As expected, retrieval becomes more difficult as the viewpoint distance increases. Interestingly, VGGT-Ω performs slightly better in the nearest quartile, while WTM outperforms it in all three more distant quartiles, with the largest R@1 improvement in Q3 (+6.06 pp).

These results suggest that WTM's learned memory is particularly useful for predicting and distinguishing viewpoints farther from the observations, rather than simply improving predictions at nearby viewpoints.

AuthorsOpenReview ↗

Q4 & W2. IVP Performance and Computational Cost (Table 3)

We thank the reviewer for raising this concern. We found that the original global search stage was unnecessarily complex. Previously, our candidate pool combined uniformly sampled global poses with locally reachable poses. However, the global candidates were sampled without spatial priors and could include many invalid or irrelevant poses (e.g., outside scene boundaries or inside objects), introducing unnecessary noise into greedy matching. In the updated submission, we remove global search entirely and retain only greedy local search with horizon (H=3). This reduces the number of candidate views from approximately 2,250 to 420 per decision, while improving WTM's IVP success rate to 56.8%, about 9 percentage points higher than the task-trained ViewAgent (47.8%).

We emphasize that the planner is only a simple interface for applying WTM's view-prediction capability to IVP, rather than a separately trained policy or a contribution of this paper. The same search procedure is also used for VGGT-Ω and FSM-LVSM-3D.

We have added inference time and TFLOPs to Table 3 in the updated submission. The complete updated table is reproduced below:

Table 3. Performance on ViewSuite: Time and TFLOPs are reported per IVP decision.

MethodParametersP2VV2PIVPAllTime (s)TFLOPs
General VLMs
GPT-5.4 Pro--53.150.718.540.8----
Qwen3.5397B39.735.54.326.5----
Qwen2.5-VL7B29.524.22.518.7----
Task-specifically Trained VLM
ViewAgent (SFT+RL based on Qwen2.5-VL)7B57.352.847.852.61.268.9
Iterative Inverse Matching
VGGT-Ω1.1B----19.1--15.6291.5
FSM-LVSM-3D0.31B----16.2--69.91985
WTM-pose (Ours)0.12B66.482.356.868.52.377.0

As for computational cost, WTM has the lowest running time and FLOPs among the three 3D models using the greedy planner. (3 times less than VGGT-Ω\Omega)

Among the VLMs, ViewAgent is trained with ViewSuite data via SFT+RL (72 hours on 8xH100 GPUs), so that its capability on this task comes from the extra training, whereas the 3D models are not specially trained on this task. Hence we put ViewAgent here as a reference, rather than a fair comparison. Considering that, we still achieve much better IVP results (56.8 vs. 47.8), with cost comparable to ViewAgent's (77.0 vs. 68.9 TFLOPs), although ViewAgent achieves lower inference latency (1.2 s vs. 2.3 s) with optimized VLM inference backends. WTM is also substantially more efficient than VGGT-Ω and FSM-LVSM-3D under the same search procedure.

We also measured the component-wise cost of WTM. DA3 and DINOv2 account for only 1.9 TFLOPs combined, while WTM's candidate-view predictions account for 74.1 TFLOPs out of the total 77.0 TFLOPs per decision.

The peak GPU memory usage during IVP inference: 21.86 GiB for WTM and 66.67 GiB for ViewAgent (SGLang). Thus, WTM requires approximately 3× less peak GPU memory than ViewAgent in our evaluated configurations.

Regarding statistical uncertainty, both WTM and our greedy planner are deterministic at inference time. We therefore independently train WTM with three different random seeds to evaluate the variability of results.

MethodIVP Success Rate (%) ↑
ViewAgent47.80
WTM (seed 7, original)56.79
WTM (seed 42)58.49
WTM (seed 52)57.92
WTM (Mean ± Std)57.73 ± 0.87

WTM consistently outperforms ViewAgent across all three training seeds, achieving an average advantage of +9.93 percentage points, with a small standard deviation of 0.87 pp. This further supports the stability of our IVP performance improvement.

AuthorsOpenReview ↗

To Reviewer 3PPz

Dear Reviewer 3PPz,

Thank you again for your time and effort for review!

We understand this is a busy period, but we would like to briefly follow up on our point-by-point responses and revised manuscript, where we have highlighted the changes and included additional experiments and clarifications to address your concerns.

We would greatly appreciate it if you could take another look and let us know whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification or discussion.

Thank you again for your time and consideration!

Best regards,

The Authors

Reviewer_SazJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes spatial memorization system to perform camera re-localization task (=IVP task, Table 3), camera pose predictions (=V2P), and nearest view search task (=I2V). Fundamentally, the method relies on the DINOv2 backbone which is frozen during training and inference. So that during optimization the memory embeddings do not become flattened or collapsed. So the memory is optimized to minimize the local / global losses.

Critical strengths

S-1. Looks interesting.

Quite interesting paper to me. The authors proposed spatial memory system where the memory is trained to learn appearance information from each scene having distinctive embeddings not to confuse with observations from different viewpoints. Nowadays, feed-forward models like VGGT, Depth-Anything-v3 are widely known, but this spatial memory system is something new to me.

S-2. Paper is well organized.

I have no difficulties reading this paper but slot routing part and Eq.8 are still quite difficult to follow. I want the authors to precisely explain this part in the rebuttal.

Critical weaknesses

W-1 Missing comparisons.

While the proposed method newly envisions the usage of spatial memory, but the target task is not something new. For example, COLMAP can also do the same tasks. Camera-2-Pose, Pose-2-Camera, Inverse View Predictions (=camera re-localization) tasks. More recently, there are some works like ACE-G and ACE. As far as my understanding both methods also can perform camera re-localization task.

On the other hand, while the authors compare the proposed method with feed-forward methods, but this method follows per-scene optimization paradigm. So, this method can show the high accuracy compared to the feed-forward methods.

It is great to provide various results with many previous studies. So that we can find the strengths of this paper in terms of experiments. I do not want to claim that State-of-the-Art is the most important factor, but showing these additional results are quite necessary in my opinion.

Based on this observation, I would say this paper provides limited experiments showing less numbers of related works. I will look forward to reading the rebuttals.

Questions for authors

Q-1. Training time

How long does it take to optimize the memory?

Q-2. Technical comparisons with "Scene Coordinate Regression"

For me, series of papers from "scene coordinate regression" are highly similar to the proposed method:

Can authors provide me the answers?

4 comments
AuthorsOpenReview ↗

W1 & Q2. Comparison with SCR & Per-Scene Clarification

We thank the reviewer for highlighting the closely related scene coordinate regression (SCR) literature, which we have already incorporated in our updated version. However, WTM differs from conventional SCR methods in two important respects: its inference procedure and its primary task.

1. WTM does not require per-scene optimization, when testing the parameters are frozen and only memory updated. WTM's Reader and Writer are trained offline and directly applied to entirely unseen test scenes, with all model parameters frozen. Instead of scene-specific training, WTM simply writes a few posed observations into its memory: 4 frames in Tab. 1 and only 1 frame in the View2Pose experiment (Tab. 3). Writing 4 frames takes only 140 ms on an A100 GPU. In contrast, R-SCoRe [4] uses approximately 4k–20k posed images per scene and requires 4–8 hours of scene-specific training on four RTX 4090 GPUs.

2. Different primary tasks and methodology. We agree that SCR is closely related to our inverse View2Pose task. However, SCR methods primarily focus on visual relocalization through scene coordinate regression and geometric pose estimation. In contrast, WTM is designed for forward view prediction (Pose2View) from limited observations, using a persistent scene memory. We further use this predictive capability for inverse View2Pose by searching for the candidate pose whose predicted features best match the query image. Although WTM is not specifically trained for visual relocalization, its learned view-prediction capability enables it to perform View2Pose through inverse matching.

3. Direct comparison under our evaluation protocol. Because conventional SCR methods generally require many mapping images to optimize a scene-specific model, their published localization results are not directly comparable to our single-reference-view, zero-shot setting. We have expanded our Related Work discussion to better clarify these differences. Moreover, our paradigm(maintains a recurrent scene representation without per-scene optimization), is also similar to CUT3R, which has been compared in Tab1. [8]

(See references in the below section.)

AuthorsOpenReview ↗

Q1 & more on W1

Q1. Training Time

As explained above, WTM does not require per-scene training. We therefore report its offline pretraining time and memory writing time instead.

A complete offline training run takes approximately 11.6 hours on eight NVIDIA A100 80GB GPUs for 30,000 optimization steps, using cached DINOv2 features. Once pretrained, the model can directly operate on unseen scenes without further optimization. Under the original FP32 configuration, memory writing takes 36.7 ms per observed image on one A100, excluding image encoding and depth estimation.

Unlike the per-scene training cost of R-SCoRe, our pretraining cost is paid only once and is therefore not directly comparable.

W1. Additional Comparisons with Classical 3D Methods

We thank the reviewer for suggesting additional comparisons with classical 3D reconstruction and camera localization methods. We agree that these are relevant approaches and have added several new baselines under our evaluation setting.

1. Evaluation setting and applicability.

First, we clarify the three tasks in ViewSuite (Tab. 3):

  • Path-to-View (P2V): Given an initial RGB image and a relative target camera pose, identify the corresponding target view.
  • View-to-Path (V2P): Given an initial RGB image and a target image, estimate the relative camera pose.
  • Interactive View Planning (IVP): Starting from an initial view, interactively select camera movements to reach a target view.

Importantly, WTM starts with only one posed RGB observation in Tab. 3. P2V and V2P are single-turn tasks, while IVP allows the agent to acquire new observations over multiple turns.

In this setting, conventional per-scene optimized methods cannot be directly applied, while standard SfM/SLAM methods typically require multiple observations to build a map. However, two-view relative-pose estimation methods can operate in our setting and provide meaningful comparisons.

2. Newly added baselines.

We implemented four approaches that can operate with a single reference view:

  • Map-free RPR [5]: Directly predicts relative camera pose from two RGB images.
  • Map-free Matching + Depth [5]: Uses image matching and monocular depth for geometric pose estimation.
  • COLMAP-SIFT + Depth + PnP [7]: Uses SIFT matching, predicted depth, and PnP to estimate relative pose.
  • MicKey [6]: A two-view method for estimating metric relative camera pose.

We adapt these relative-pose estimates to P2V and V2P under the same evaluation protocol as WTM, without scene-specific training.

MethodP2V (%)V2P (%)
Map-free RPR37.7435.47
Map-free Matching + Depth33.0231.70
COLMAP-SIFT + Depth + PnP38.1139.62
MicKey33.7731.70
WTM-pose (Ours)66.4282.26

WTM outperforms all four baselines on both tasks. These results complement our existing comparisons with SfM/SLAM, feed-forward 3D reconstruction, and novel-view synthesis methods in Tab. 1.

For IVP, the task additionally requires choosing actions over multiple turns, which is not directly addressed by a localization method alone. We therefore also compare with iterative inverse-matching approaches (VGGT-Ω\Omega and FSM-LVSM-3D) using the same search procedure, as reported in Tab. 3.

This table has been added into appendix in the updated submission, providing more directly comparable baselines. We have also expanded the Related Work discussion to better position WTM relative to scene coordinate regression [1–4] and classical visual localization methods.

References

[1] Brachmann et al. Accelerated Coordinate Encoding: Learning to Relocalize in Minutes Using RGB and Poses. CVPR 2023.

[2] Bruns et al. ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Training. ICCV 2025.

[3] Shotton et al. Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images. CVPR 2013.

[4] Jiang et al. R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization. CVPR 2025.

[5] Arnold et al. Map-free Visual Relocalization: Metric Pose Relative to a Single Image. ECCV 2022.

[6] Barroso-Laguna et al. Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences. CVPR 2024.

[7] Schönberger and Frahm. Structure-from-Motion Revisited. CVPR 2016.

[8] Wang et al. CUT3R: Continuous 3D Perception Model with Persistent State. CVPR 2025.

AuthorsOpenReview ↗

S2. Clarification of Eq. 8

Thanks for the positive comments on the clarity of our paper! We are happy to further explain Eq. 8, which involves one of our core ideas: using the same routing mechanism for memory reading and writing.

The key question is: when a new observation arrives, how do we decide which memory slots should receive its information, and how much each slot should receive?

Our intuition is simple: the slots that contribute to predicting a patch should also incorporate new evidence from that patch. We therefore reuse the Reader’s attention weights to assign responsibility for writing. Different slots learn to represent different parts of the scene. If a patch strongly attends to a particular slot during reading, that slot should also be responsible for incorporating new information from that patch.

The quantities involved are:

  • L: Number of memory slots.
  • N: Number of image patches.
  • Rt∈RN×LR_t\in\mathbb{R}^{N\times L}: The pre-write attention weights retained from the Reader. Rt[n,l]R_t[n,l] indicates how much slot ll is attended to when predicting patch nn, or intuitively, how much this slot contributes to the prediction of that patch.
  • rt∈RLr_t\in\mathbb{R}^{L}: The column-wise sum of RtR_t, i.e., rt[l]=∑nRt[n,l]r_t[l]=\sum_n R_t[n,l]. We use it for normalization. Because the basic unit of our computation and memory update is a patch, and we do not want the update magnitude to grow simply because an image contains more patches (larger (N)).
  • E∈RN×DE\in\mathbb{R}^{N\times D}: The evidence we want to write into memory. It takes different forms for Content and Address updates, so here we can simply view (E[n,:]) as the evidence associated with patch n.

Specifically, before writing an observation, we first perform a pre-write read and retain the attention weights Rt∈RN×LR_t\in\mathbb{R}^{N\times L}, where NN is the number of image patches and LL is the number of memory slots. Given patch-level evidence E∈RN×DE\in\mathbb{R}^{N\times D}, we route it back into the slots using:

Route⁡t(E)=Diag⁡(rt+ϵ)−1Rt⊤E,rt=Rt⊤1N.\operatorname{Route}_t(E) =\operatorname{Diag}(r_t+\epsilon)^{-1}R_t^\top E, \qquad r_t=R_t^\top\mathbf{1}_N.

Here, Rt⊤ER_t^\top E aggregates the patch-level evidence into memory slots, while rtr_t is the total attention mass received by each slot. Dividing by this mass normalizes the aggregated evidence, so that its magnitude does not simply grow with the number of contributing patches.

This is the operation described in Eq. 8. The evidence EE takes different forms for Content and Address updates (Sec. 3.4), but both use the same routing mechanism, connecting how the memory is read with how it is updated.

AuthorsOpenReview ↗

To Reviewer SazJ

Dear Reviewer SazJ,

Thank you again for your time and effort for review!

We understand this is a busy period, but we would like to briefly follow up on our point-by-point responses and revised manuscript, where we have highlighted the changes and included additional experiments and clarifications to address your concerns.

We would greatly appreciate it if you could take another look and let us know whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification or discussion.

Thank you again for your time and consideration!

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Grateful thanks to all reviewers and a brief summary of our updated submission

We thank all reviewers for their careful reading and constructive feedback. We appreciate their recognition of the technical value of WTM’s learned read–write memory and its shared interface for forward view prediction and inverse viewpoint matching.

The reviewers’ comments have helped us strengthen the experimental evaluation and clarify the scope of our contributions.

We also received several valuable concerns and suggestions, which we have carefully considered and addressed. To reflect these improvements, we have uploaded a revised version of our paper. The main changes are:

1. (To all three reviewers) Simplified planner in Table 3 (IVP), with stronger performance and substantially lower computational cost. Thanks to the reviewers' suggestions on the planner, we carefully revisited our design and found that the original global search stage was unnecessarily complex. Previously, the candidate pool consisted of a global set (uniformly sampled poses across the scene) and a local set (poses reachable from the current position within (H=3) or (4) actions). By simply removing global search, adopting a shorter horizon and keeping only greedy local search, the IVP success rate improves from 48.5% to 56.8%, with only 420 candidates per decision round. This improvement is intuitive: the global set is sampled without spatial priors and contains many invalid or irrelevant poses (e.g., outside scene boundaries or inside objects), introducing unnecessary noise into greedy matching. Simple short-range imagination and greedy search is just sufficient for view planning task. The reviewers' suggestions helped us recognize this issue and simplify the planner, a nice example of Occam's razor in practice.

2. (To all three reviewers) Added computational cost comparisons in Table 3. We have added an additional column reporting computational costs, making it clear that our performance on the ViewSuite task does not rely on an unfair computational advantage.

3. (To reviewer SazJ) Expanded the discussion of closely related methods in Related Work. This helps better position our work.

Other concerns and questions raised by the reviewers are also addressed point by point below. Thanks again for your time and valuable feedback!

AuthorsOpenReview ↗

Thanks and Final Comment to our reviewers and AC

For our reviewers

We sincerely thank all reviewers for their time and effort in reviewing our submission. Your comments and discussions have helped us strengthen our work by better positioning our method relative to prior work (Reviewer SazJ), clarifying the contribution of our read–write mechanism through additional ablations (Reviewer 3PPz), and simplifying our planner and optimizing inference to achieve lower latency and computation cost than the evaluated baselines (Reviewers 3PPz and 5rMr).

For the AC

We sincerely thank you for your time and effort in handling our submission, especially for your post-rebuttal comment summarizing the reviewers' key concerns and encouraging further discussion.

However, we would like to raise a concern regarding Reviewer 5rMr's follow-up assessment.

As you highlighted in your summary, the reviewer's original concerns centered on the geometry-aligned feature memory baseline and the computational fairness of the ViewSuite evaluation. In fact, the reviewer explicitly identified the former as "additional evidence that could materially change my assessment." We conducted the requested experiments and substantially improved the evaluation. The reviewer acknowledged that these experiments "largely address my original concern".

However, despite this acknowledgment, Reviewer 5rMr shift their position and the emphasis of their assessment toward "limited practical significance" and the lack of "sufficiently compelling new capabilities" in realistic embodied settings. We find this difficult to reconcile with the initial review, which explicitly recognized WTM as "a technically meaningful design" and stated that its broad validation "makes the work more consequential than a single-benchmark architectural result." The follow-up does not explain why these previously acknowledged contributions are now considered insufficient, nor does it specify what additional evidence could address this newly emphasized concern.

We fully respect the reviewer's decision and judgment. Our concern, however, is not the score itself, but the consistency and fairness of the evaluation. The shift in assessment rationale, together with the introduction of new concerns without clear supporting evidence after acknowledging that the original concerns were largely resolved, raises serious questions about the consistency and fairness of this review.

Given these circumstances, we have decided to withdraw our submission.

Once again, we sincerely appreciate the time and effort you have devoted to handling our submission. We wish you all the best with the remaining submissions in your AC batch.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.