- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LOCI, a hybrid spatial-memory architecture for camera-controllable streaming world models. It combines explicit historical KV access with a fixed-size recurrent KDA memory. The recurrent path is conditioned on camera geometry through PRoPE, and its readout influences later historical-attention layers. The method also introduces per-chunk retention and a bounded bank of retained observations for long-horizon streaming. Experiments on MIND and held-out trajectories show better revisit fidelity than a same-recipe full-softmax baseline, including under the same bounded history budget, while reducing memory usage and enabling longer generation.
Critical strengths
- The controlled comparison with full softmax is convincing. The two models use the same backbone, training data, recipe, and number of updates, and the bounded setting also uses the same retained observations. The consistent gains therefore provide reasonably strong evidence that the overall hybrid memory design is useful.
- The paper evaluates both memory quality and practical efficiency. Besides revisit fidelity, the paper analyzes long-range recall, recurrent-state behavior, historical attention, and memory/runtime cost. The bounded version supports long streaming generation with essentially constant memory while maintaining better revisit fidelity than the corresponding full-softmax baseline.
Critical weaknesses
- The contribution of the key architectural components is not sufficiently isolated. The main controlled comparison changes several factors at once, including the recurrent KDA path, the distribution of local versus historical-attention layers, additional parameters, and PRoPE conditioning. In particular, the paper does not report end-task revisit metrics for an otherwise matched recurrent model without PRoPE. The yaw-decoding analysis mainly shows that camera information is encoded in the state, but does not establish how much projective conditioning contributes to the final generation quality.
- The long-horizon streaming claim is not fully validated under the deployment setting. During training, recurrent states are formed by scanning noised ground-truth chunks, whereas at inference they are persistently committed from the model’s own generated chunks after denoising. This train–inference mismatch is directly relevant to long-horizon memory, where generation errors may accumulate in the state. The existing rollout results are encouraging, but the effect of the commit strategy is not directly ablated, and the empirical scope is further limited to one 5B backbone and mostly rendered held-out trajectories.
Questions for authors
- Could the authors report end-to-end MIND and/or held-out revisit results for the trained recurrent variant without PRoPE camera conditioning? This would directly clarify how much of LOCI’s gain comes from projective conditioning itself rather than from the recurrent hybrid architecture more generally.
- Could the authors quantify the impact of the train–inference mismatch in recurrent-state construction? In particular, how sensitive are long-horizon results to the commit strategy and the choice of t=100 in the bounded setting, and does performance change when memory is accumulated from generated rather than clean/noised history?
3 comments
Thank you for the detailed response and additional experiments.
The generated-history rollouts and commit-timestep controls largely address my main concerns. The half-softmax comparison also helps clarify the contribution of the recurrent memory.
My remaining concerns are mainly about the limited independent benefit of PRoPE under the original UCPE setting and the lack of reliable real-world revisit evaluation. However, the authors have clarified these limitations, and the new results provide stronger support for the overall hybrid design.
I appreciate the revisions and maintain my weak-accept rating.
We sincerely appreciate your positive assessment and the time you spent reviewing our paper and responses. Your questions regarding the generated history and the commit timestep prompted valuable experiments that strengthened the revised paper, and we will continue to refine it for the final version.
Thank you for your thoughtful review. Following your questions, we added experiments on commit sensitivity, generated-history effects and long-horizon revisit fidelity. The global response summarizes the shared results; intervals below are paired cluster-bootstrap 95% CIs.
W2 / Q2 — train–inference mismatch, commit strategy, generated history. We agree that training scans noised ground-truth chunks in one forward pass, whereas inference commits the model's own denoised chunk in a separate pass. In our tests, committing with light noise (t=100) helps both models, and LOCI keeps its revisit advantage at both commit timesteps and with generated history.
These tests change KV history and recurrent state together; they do not isolate recurrent-state mismatch alone. This and the single 5B backbone remain limitations. Beyond rendered trajectories, generating from 10 real photographs, LOCI still leads (+0.96 dB [+0.64, +1.30], 49/60 cases; App. G.5, Table 20); recorded real-video revisits remain open.
W1 / Q1 — PRoPE versus the recurrent hybrid architecture. The evidence attributes the gain over full softmax to the complete hybrid design. With UCPE, PRoPE helps early in training, but by 5,000 updates there is no consistent benefit (revisit LPIPS +0.009 [−0.001, +0.020] on 60 designed cases; −0.004 [−0.016, +0.006] on held-out recorded set A, bounded). Because UCPE already carries camera information into the state, we also replaced it with a frame-level embedding; there, adding PRoPE gives −0.018 revisit LPIPS [−0.035, −0.004] at 5,000 updates (37/60; one run per arm). These comparisons do not separate PRoPE's contribution at convergence (global G2 and App. G.3, Table 18 give all checkpoints).
To isolate the split, we trained a half-softmax control with LOCI's 15 chunk-local layers and no state. At the same 2,000-update budget, on two separate evaluation sets of 60 revisits from different maps (same checkpoints), it is not significantly different from full softmax (+0.004 and −0.015 LPIPS; both CIs include 0). LOCI improves over this control by −0.071 [−0.096, −0.049] and −0.048 [−0.065, −0.034] LPIPS (50/60 and 49/60), so at 2,000 updates the layer split alone does not account for the gain (App. G.4, Table 19).
The recurrent branch adds 90.7M parameters (1.7% of 5.38B); the comparison is not parameter-matched, and alternative layer arrangements were not evaluated. We have revised the abstract, introduction and contributions to reflect this attribution, and frame Table 2 as a comparison of the complete hybrid design against the standard design under identical backbone, data, schedule and bounded budget.