acceptodds
Withdrawn from ICLR 2027

Geometry Tells Where; History Tells What: Appearance Memory in Causal Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generators now demonstrate impressive capabilities in creating controlled qualitative content. However, camera-controlled video generators still produce substantial appearance drift when revisiting earlier viewpoints. In this paper, we build a competitive geometry- and camera-conditioned causal backbone and use it to investigate what information should be preserved and where recalled memory should enter the generator. We find that geometry alone is insufficient, even when it covers almost the entire view: appearance is recovered reliably only when previously generated RGB image is recalled and temporally aligned with the frame currently being generated. Crucially, simply providing the correct recalled RGB is not enough: when its temporal position is reset to the beginning of the sequence, rather than current ,its benefit largely disappears, performing no better than having no memory. Based on these findings, we introduce CHaRM, a training-free appearance memory that stores generated RGB frames indexed by camera pose. At revisit time, CHaRM reuses the stored frames when the camera returns to a stored pose, or otherwise warps nearby stored frames into the new view, gives them the temporal position of the frame being generated, and injects them next to the KV cache of recent frames. As a result, CHaRM cuts self-revisit LPIPS (consistency with previously generated views) from 0.504 to 0.099 on exact revisits, and brings return LPIPS (fidelity to ground truth on the way back) to the level the backbone reaches in a single uninterrupted pass, i.e., with no loss of quality from revisiting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera Controlled, Causal Autoregressive Video Generation, Geometry Conditioned, Appearance Memory, Temporal Addressing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zttcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies appearance drift when revisiting previously observed views in camera-controlled causal video generation. To address this issue, it proposes CHaRM, a training-free memory mechanism. The method stores RGB frames generated during the initial visit along with their camera poses. Upon revisiting, it either directly retrieves matching frames or reprojects nearby historical frames into the target view. The retrieved content is encoded into keys and values (KV), assigned temporal positional encodings aligned with the current generation chunk, and used alongside the existing historical KV cache during generation. Experiments show that the method improves appearance consistency upon revisiting in the evaluated scenarios.

Critical strengths
  1. The research problem is practically meaningful. Maintaining appearance consistency when the camera returns to previously visited locations is an important requirement for building generative environments that support continued exploration.

  2. As a training-free method applied to a frozen backbone, CHaRM achieves strong results on the reported revisitation benchmarks. However, only camera control and 3D consistency are reported for WorldScore, which is insufficient for a comprehensive assessment of generation quality.

Critical weaknesses
  1. The paper offers limited novelty. The idea of using point clouds to provide geometric information and reference images to supplement appearance details has been extensively explored in prior work. For example, Spatia[1] uses point clouds to provide geometry and historical frames to provide RGB details.

  2. The ablation is insufficient. First, the necessity of warping reference images is unclear. It remains uncertain how much improvement warping provides over directly using historical RGB frames, especially given that it introduces additional holes. A direct comparison between warped RGB frames and the original historical RGB frames under identical settings is needed to establish the actual benefit of this step. Second, the ablation study on temporal positions is limited. The current experiments primarily compare resetting RoPE positions to zero with aligning them to the current generation chunk, without considering alternatives such as retaining the historical frames’ original temporal positions.

[1] Spatia: Video Generation with Updatable Spatial Memory

Questions for authors
  1. Where is the appendix?
  2. Can you report results for all WorldScore metrics?
Reviewer_GGuNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies appearance drift when a camera-controlled causal video generator revisits earlier viewpoints. It introduces CHaRM, which stores generated images indexed by camera pose, retrieves or warps them into the requested view, and injects their representations alongside the persistent KV cache. Its central finding is that recalled images are substantially more effective when assigned the temporal positions of the frames currently being generated.

Critical strengths

Strength 1: The matched temporal-position experiment is compelling: keeping recalled RGB and other inputs fixed, target alignment reduces self-revisit LPIPS from 0.485 to 0.097 compared with resetting positions to zero.

Strength 2: CHaRM substantially improves exact-revisit consistency without additional training of the frozen backbone, while maintaining similar average fidelity to real ground truth.

Critical weaknesses

Weakness 1: The methodological novelty is limited by Warp-as-History, which already uses target-aligned pseudo-history and visibility filtering. The additional contribution of persistent-KV integration needs clearer justification.

Weakness 2: Shifted-revisit evaluation uses warped references closely related to the conditioning inputs. These proxy metrics cannot independently establish target-view accuracy, particularly in disoccluded regions.

Questions for authors

Question 1: Can the authors evaluate shifted revisits against independently captured or rendered target-view ground truth, reporting results separately for visible and disoccluded regions?

Question 2: How does CHaRM compare with replaying stored images while refreshing the generator’s KV cache through re-encoding or teacher forcing, particularly when departing toward unseen viewpoints?

Reviewer_eUFqOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors build a 1.3B camera- and geometry-conditioned backbone, then freeze it for memory experiments. CHaRM stores generated frames with camera poses. It recalls a matching frame, or warps a nearby frame, and adds its keys and values beside the causal cache. The central experiment changes only the temporal position of recalled content: using the current target position works much better than resetting it to zero. Across 55 DL3DV scenes, exact-revisit self-consistency LPIPS improves from 0.504 to 0.099, while return LPIPS against real frames improves from 0.600 to 0.445. Further tests use shuffled revisit order and shifted poses. The clearest contribution is the controlled temporal-position finding on this backbone.

Critical strengths

1: The design gives geometry and appearance memory clear roles. Geometry describes where surfaces are, stored RGB frames preserve their earlier appearance, and the causal cache maintains continuity between nearby frames. This separation makes the method easy to understand and gives each component a clear purpose.

2: The paper treats temporal alignment as part of memory design. Retrieving the right image is only one part of recall. The method also specifies where the recalled content should enter the causal sequence. This provides a concrete design principle for making earlier observations useful during current generation.

3: The retrieval strategy handles both exact and nearby revisits. Stored images are used directly when camera poses match. For nearby viewpoints, the method warps a stored image and checks visibility and coverage before using it. These explicit rules help avoid injecting unsuitable history.

4: The memory module preserves the generator’s existing context. CHaRM adds recalled keys and values alongside the ordinary causal cache, so recent history remains available. The memory module requires no additional training of the frozen backbone, making it a modular extension.

Critical weaknesses

1: The stored camera poses may not match the generated views. Memory is indexed by camera commands rather than poses estimated from the generated images. If the generator does not follow a command accurately, a stored image and its pose may disagree. This could affect retrieval and warping. Testing sensitivity to camera-following errors would clarify how reliable this assumption is.

2: The frozen memory cannot correct earlier mistakes. Keeping the bank unchanged protects it from accumulating errors through repeated updates. However, it also prevents replacing a poor first-visit frame with a better later observation. The method therefore favors preserving earlier appearance, while how to correct stored artifacts or inconsistent details remains unclear.

Questions for authors

1: With the same KV injection route, token budget, and temporal position, how do original RGB, nearest-source warps, and merged point-cloud renderings compare?

2: Can shifted revisits be evaluated with independently rendered ground-truth query views? Please also report coverage and rejection rates as camera displacement and disocclusion increase.

3: What is the added latency per generated block, including reconstruction, VAE encoding, and KV capture, for direct and warped recall? How does performance change with larger history banks or longer trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.