acceptodds
Withdrawn from ICLR 2027

TraceWorld: Trajectory-Indexed Memory for Spatially Consistent Video World Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Camera-controlled video world models aim to simulate coherent visual environments across changing viewpoints. When the camera revisits a previously observed region, maintaining this coherence requires scene information that may no longer be available in recent temporal context. We introduce TraceWorld, a video world model with view-adaptive key-value (KV) memory allocation for camera-controlled generation. To incorporate camera geometry while preserving the pretrained visual pathway, TraceWorld adds a decoupled camera-aware attention branch. Pose-affinity retrieval then selects historical latent frames and reuses their cached KV states across both attention branches. A learned allocation policy divides a fixed context budget between recent and retrieved KV states, conditioned on inter-chunk camera-pose changes and the pose affinity between target views and stored history. Together, these components balance local temporal continuity with scene recall without constructing an explicit 3D scene memory. To evaluate scene consistency under camera revisits, we introduce the Scene Recall Benchmark (SRBench), comprising 200 camera-return cases across eight motion categories. On WorldScore, TraceWorld achieves the highest object control and 3D consistency scores among the compared methods. On SRBench, it achieves the highest revisit consistency and margin while remaining competitive in stability and camera following.

Primary area:
generative models
Keywords:
video generation, video world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GRfEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TraceWorld, an autoregressive camera-controlled video world model that maintains spatial consistency across revisited viewpoints without an explicit three-dimensional scene representation. The method augments a pretrained video diffusion transformer with a downsampled projective positional encoding attention branch for query-independent key-value caching, alongside a learned policy that dynamically splits a fixed historical key-value budget between recent frames and past frames retrieved by camera-pose affinity. Evaluations on WorldScore and a newly proposed two-hundred-case Scene Recall Benchmark show strong gains in revisit and geometric consistency over general video generators and point-cloud rendering baselines, with ablations validating both the retrieval and adaptive allocation mechanisms.

Critical strengths
  1. Query-Independent Camera-Aware KV Caching without Explicit 3D Reconstruction:

Augmenting the pretrained visual pathway with a 4x spatially downsampled projective positional encoding (PRoPE) branch allows historical keys and values to be transformed solely by their source camera matrices, while target camera matrices are applied only to queries and outputs. This makes cached camera-aware KV states strictly query-independent and reusable across arbitrary future viewpoints without per-step 3D depth estimation, point-cloud fusion, or rendering artifacts, driving strong gains in WorldScore 3D consistency (93.40 vs. 88.51 for the best baseline in Table 1) and SRBench revisit consistency (0.6979 in Table 2).

  1. Rigorous Revisit Evaluation via Control-Pair Margin:

SRBench pairs Revisit Consistency (RCS) with a deterministic Margin metric computed against temporally matched, pose-separated control pairs. Because degenerate slow-moving or static videos artificially inflate both raw frame similarity and adjacent-frame PSNR stability, evaluating similarity margins over matched temporal gaps is consequential for proving genuine viewpoint-specific scene recall (improving Margin from 0.0988 with recent-only KV to 0.2086 with adaptive pose-affinity allocation in Table 3).

Critical weaknesses
  1. Horizon Mismatch Between the 81-Frame Benchmark and the C=12C=12 Latent-Frame Budget:

With 3 VAE latent timesteps per chunk, the 81-frame SRBench horizon spans 21 latent frames (7 chunks) with the initial frame held separately as an attention sink. Because adaptive allocation is bypassed whenever the eligible history size is at most C=12C = 12 (Appendix 6.4), the policy is inactive during the first 5 of 7 chunks (11 or fewer eligible historical frames, covering the first 61 RGB frames) and executes only twice per video (at chunks 5 and 6, with 14 and 17 eligible frames). Moreover, since SRBench trajectories reverse around the midpoint, the model is already in the return phase whenever the history exceeds 12 frames; thus, SRBench never quantitatively tests whether the policy preserves recent context during long outward exploration without relevant history, nor are quantitative metrics, Full-KV upper bounds, simple threshold baselines on historical affinity, or learned retrieval-size distributions reported for long rollouts (where 484-frame examples are only shown qualitatively in the appendix).

  1. Omission of Key Memory-Augmented World Model Literature and Confounded Baseline Comparisons:

The related work and conceptual comparison omit directly related memory-augmented video world models—most notably WorldPack (Oshima et al., 2025, TMLR), WorldMem (Xiao et al., NeurIPS 2025), VMem (Li et al., ICCV 2025), and SlowFast-VGen (Hong et al., ICLR 2025). WorldPack already combines recent trajectory context packing with camera-pose-guided memory retrieval under a bounded context budget for revisit consistency, and WorldMem already retrieves pose-indexed memory via camera position/orientation similarity and pose-aware attention without explicit 3D reconstruction; omitting them overstates TraceWorld's conceptual novelty. Empirically, Tables 1 and 2 exclude all prior video-memory world models (including WorldPack, WorldMem, and CaM), and Table 1 compares TraceWorld (fine-tuned from Wan2.2-5B on 1,230 hours of camera-annotated video) against off-the-shelf Wan2.1 without reporting the memory-disabled Stage-1 backbone (Wan2.2-5B + PRoPE) on WorldScore, confounding backbone and data gains with memory allocation.

Questions for authors
  1. Quantitative Evaluation and Policy Behavior on Long Rollouts:

In the 81-frame SRBench evaluation, the eligible history exceeds the budget C=12C = 12 only in the last two chunks (chunks 5 and 6), where the camera is already returning. Can you report quantitative SRBench metrics (including the Table 3 ablations, a Full-KV upper bound, and a simple threshold heuristic on historical pose affinity) on longer trajectories (e.g., 200+ frames or multi-loop trajectories) where the history substantially exceeds CC during both outward exploration and revisit phases, along with the empirical distribution of predicted retrieval sizes across chunks?

  1. Comparison with Omitted Memory-Augmented World Models and Stage-1 Backbone Ablation:

Can you clarify the conceptual and empirical differences between TraceWorld and prior geometry-free memory world models omitted from the related work—specifically WorldPack (Oshima et al., 2025, TMLR), which combines recent trajectory packing with camera-pose-guided memory retrieval under a fixed budget, and WorldMem (Xiao et al., NeurIPS 2025), which retrieves pose-indexed frames via position/orientation similarity and pose-aware attention? Furthermore, how do WorldPack, WorldMem, CaM, and your memory-disabled Stage-1 backbone (Wan2.2-5B + PRoPE trained on the same camera-annotated corpus) perform on WorldScore (Table 1) and SRBench (Table 2)?

  1. Prefix Evaluation of Shorter Outputs on SRBench:

The paper states that shorter baseline outputs are evaluated on their available prefix without temporal padding, whereas revisit pairs are defined on the fixed 81-frame trajectory. Which baselines in Table 2 produced fewer than 81 frames, and how were post-turning revisit pairs exceeding a model's output length handled when computing RCS and Margin?

Reviewer_ndX4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TraceWorld, a camera-controlled autoregressive video generation model designed to preserve scene content when revisiting previously observed viewpoints. Built on Wan2.2-5B, it combines a decoupled PRoPE-based camera-aware attention branch with retrieval of historical, branch-specific KV states. A learned allocation policy uses camera-motion and pose-affinity statistics to divide a fixed active history budget between recent and retrieved latent frames. The paper also introduces SRBench, comprising 200 camera-return cases across eight motion categories. Experiments on WorldScore and SRBench show competitive generation quality and improved revisit consistency, while ablations demonstrate benefits from combining recent and retrieved context.

Critical strengths
  1. The memory design addresses a concrete tension between temporal continuity and scene recall. Recent frames and geometrically relevant older observations provide complementary information. The same-budget ablations in Table 3 support this motivation: combining both sources improves over recent-only and retrieval-only variants, and adaptive allocation further improves all four reported metrics over the fixed 6+6 allocation. Reusing native KV states also provides a coherent way to incorporate historical evidence into the generator.

  2. The evaluation explicitly distinguishes scene recall from generic visual stability. SRBench evaluates camera-return trajectories and includes a margin against temporally matched, pose-separated control pairs. This is useful because high frame similarity alone can reward insufficient motion. Reporting recall, margin, stability, and camera following separately makes the results more informative than a single aggregate consistency score.

Critical weaknesses
  1. The evidence does not sufficiently establish the added value of the learned allocation policy, which is the central methodological contribution. Table 3 compares adaptive allocation with one fixed mixture, using six recent and six retrieved frames. Adaptive allocation improves RCS from 0.6763 to 0.6979 and Margin from 0.1842 to 0.2086, but there is no sweep over fixed mixtures or comparison with a simple rule using the same motion and retrieval-relevance statistics. Consequently, it remains unclear how much improvement specifically requires learning the allocation policy. The paper also provides no uncertainty estimates or repeated-run results; these would help interpret both the allocation gains and the relatively small RCS advantage over Spatia, 0.6979 versus 0.6882. Given that the other components build on established techniques, stronger evidence for this central contribution is important to the acceptance case.

  2. Long-horizon effectiveness and practical memory costs are insufficiently characterized. The main quantitative SRBench evaluation is capped at 81 RGB frames, while the 484-frame rollouts are evaluated qualitatively. Moreover, bounding the active context to 12 latent frames does not bound the stored KV archive: both cumulative storage and retrieval scoring grow with rollout length, as the paper acknowledges. There are no measurements of archive size, peak GPU memory, retrieval latency, or generation throughput as history grows. The current evidence therefore leaves open how well scene recall persists after substantially longer absences and at what computational cost. This is an evaluation gap rather than a contradiction of the stated fixed active-context budget.

Questions for authors
  1. Does adaptive allocation consistently outperform the best fixed recent/retrieved mixture selected on a separate validation set, and a simple allocation rule based on the same geometric statistics? Please provide uncertainty estimates for the key comparisons and, if available, the distribution of selected retrieval sizes during exploration and revisits.

  2. How is the complete historical KV archive stored during inference, including its device placement and any offloading or eviction? Can you report quantitative revisit performance together with archive size, peak GPU memory, and per-chunk latency at increasing rollout lengths, including the 484-frame setting?

  3. In the visual branch, are cached keys stored before or after temporal RoPE, and do retrieved frames retain their original temporal position IDs or receive reassigned IDs? Appendix 6.3 specifies temporal sorting before gathering, but does not resolve this positional-encoding detail, which is relevant to reproducing retrieval across large temporal gaps.

Reviewer_fbuoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TraceWorld, a camera-controlled video world model designed to improve scene consistency when the camera revisits previously observed regions. The method stores historical KV states and retrieves relevant frames based on camera pose affinity. A learned allocation policy determines how to divide a fixed attention budget between recent and retrieved frames. The framework also introduces a decoupled camera-aware attention branch based on PRoPE, avoiding the need for an explicit 3D scene representation.

The authors introduce SRBench, consisting of 200 camera-return cases, to evaluate scene recall. Results on WorldScore and SRBench show competitive generation quality and improvements in revisit consistency. Ablation studies further examine the contributions of pose-affinity retrieval and adaptive memory allocation.

Critical strengths
  1. The idea of reusing historical KV states and allocating context based on camera motion is well motivated. The separation between the visual and camera-aware attention branches is also a reasonable design choice. I find the approach interesting because it addresses scene recall within the existing attention mechanism rather than introducing an additional 3D representation.

  2. Useful ablations of the memory design. Table 3 compares recent-only context, different retrieval strategies, and fixed versus adaptive allocation under the same active memory budget. This provides some evidence that adapting the balance between recent and retrieved context is beneficial.

Critical weaknesses
  1. The evaluation does not sufficiently establish long-term scene consistency.

My main concern is the design of SRBench. The quantitative evaluation is limited to 81-frame sequences, approximately five seconds at 16 fps, and the camera trajectories explicitly reverse their earlier paths. This makes the revisit setting relatively predictable compared with longer exploration involving multiple revisits or imperfectly overlapping viewpoints.

The method also permanently retains the first latent frame as an attention sink. Since the benchmark trajectories return toward their starting viewpoints, it would be useful to understand how much this design contributes to the reported results. The 484-frame examples in Appendix 6.6 are encouraging, but they are only qualitative. I am therefore not fully convinced that the reported improvements extend to the long-horizon setting that motivates the paper.

  1. The advantage over the closest alternatives is not yet sufficiently established.

Although TraceWorld achieves the best results on SRBench, the improvement over Spatia is relatively small: 0.6979 versus 0.6882 in revisit consistency, and 0.2086 versus 0.1987 in margin. Without per-case variation or uncertainty estimates, it is difficult to judge how robust these differences are across the benchmark.

Table 3 supports the benefit of learned allocation over a fixed 6+6 split. However, it remains unclear whether a simpler rule-based allocation using the same pose-affinity scores could achieve similar results. The FOV-overlap ablation is also not a direct comparison with the complete CaM method, despite CaM being closely related to the proposed memory retrieval approach.

Taken together, these limitations make it difficult for me to assess how much improvement can be attributed specifically to the proposed adaptive allocation strategy.

Questions for authors
  1. How consistent are the improvements across SRBench cases? Could the authors report per-category results and confidence intervals for revisit consistency and margin, particularly for the comparisons with Spatia and fixed allocation? Also, are the metrics computed over identical valid revisit pairs for all methods when some generated sequences are shorter than 81 frames?

  2. How much does the method benefit from the first-frame attention sink? Would the gains persist for longer trajectories with non-exact returns or multiple revisits? Even a smaller quantitative evaluation using the extended trajectories in Appendix 6.6 would help clarify this.

  3. Is the learned allocation necessary compared with a simple heuristic? For example, how would a threshold-based allocation using historical pose-affinity scores perform under the same budget? Also provide the memory and inference-time costs as the stored KV history grows, since the fixed active context budget does not bound total KV storage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.