- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper tackles the problem of long-horizon, camera-controlled video reshooting. The authors propose to autoregressively generate, chunk by chunk, a novel-view video that follows a user-specified camera trajectory, lasts up to a few minutes, and remains synchronized with the source video. The approach formulates reshooting as a query of a dynamic memory along three axes: time (retrieving synchronized source chunks), space (spatial retrieval exploiting multi-time coverage of the same 3D point), and semantics (retrieving semantic anchors). The retrieved patches are concatenated with the noise tokens. Two correspondence embeddings are then added to the queries and keys (but not to the values): (1) a 4D correspondence embedding combining time and projected geometry, and (2) a semantic correspondence embedding based on TIPS descriptors.
The paper also introduces WorldReplay, a pipeline built with Unreal Engine 5 that produces one-minute, 8K synchronized multi-camera videos. The resulting dataset addresses the lack of long videos with synchronized multi-camera trajectories. Experiments show improvements over Vista4D, InSpatio-World, World-in-World, and CameraAnything in synchronization, camera control, and long-range consistency.
Critical strengths
The problem tackled in this paper is well motivated. The proposed solution, which decomposes long-horizon reshooting into temporal synchronization, spatial consistency, and semantic consistency, is intuitive and provides a useful conceptual framework for this problem.
The proposed approach simultaneously addresses several shortcomings of previous methods: large baselines, subjects that leave and re-enter the view, revisits, and accumulated camera drift. The paper also clearly explains how the approach resolves each of these issues.
If released, WorldReplay will be a useful dataset contribution for the community, especially if the authors also release the generation code so that it can be extended with larger scenarios and longer videos.
The paper includes meaningful ablations and shows strong improvements over the compared methods.
Critical weaknesses
Some close related works are missing:
- Park et al. "Redirector: Creating any-length video retakes with rotary camera encoding." CVPR 2026.
- Xu et al. "RealCam: Real-Time Novel-View Video Generation with Interactive Camera Control." arXiv preprint arXiv:2605.06051 (2026).
- Paliwal et al. "Reshoot-anything: A self-supervised model for in-the-wild video reshooting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
- Qiao et al. "Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks." arXiv preprint arXiv:2606.15534 (2026).
Retrieval design contains many heuristics: The framework relies on a number of heuristic design choices, for example: using only 3 keyframes for spatial retrieval, selecting 8 semantic anchors, the greedy coverage strategy for spatial selection, and the use of a VLM/LLM to choose the anchors. These choices appear somewhat arbitrary, the paper does not justify them, and there is no study of how sensitive the results are to changes in these parameters.
Novelty with respect to other memory-based world models is not sufficiently discussed. Several recent works are closely related:
- Chen et al. "Out of sight but not out of mind: Hybrid memory for dynamic video world models." CoRR, abs/2603.25716, 2026b also propose a hybrid memory for dynamic scenes and introduce a synthetic dataset similar to the one presented here.
- Xu et al. "UCM: Unified modeling of camera control and memory with time-aware positional encoding warping for world models." Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers. 2026 introduce a time-aware positional encoding warping that appears close to the 4D correspondence embedding proposed in this paper.
The related work section mostly lists prior papers without clearly explaining how the proposed method differs from them. For instance, how does the 4D embedding differ from UCM’s positional encoding warping? And how does the proposed memory differ from the hybrid memory of Chen et al.? In its current form, the paper reads more as an effective combination of existing ideas, applied to a new task and supported by a new dataset, than as a fundamentally new method. This is a legitimate and useful contribution, but the authors should state it clearly.
Presentation issues and typos:
- Fig. 1: "Input Video of an Dynamic Scene"
- Fig. 2 labels the module “Time-Aware Correspondence,” while the caption text calls it “Dynamic-Aware Correspondence.”
- Fig. 2 is very dense and could be split into two figures
- The limitations are not discussed in the main paper, but in the supplement materials.
Questions for authors
-
In the Appendix: Table 2’s full-model synchronization numbers (99.33 / 40.83 / 6.5 / 7.4 / 98.6 / 98.8) differ from Table 3’s LongReplay row (99.34 / 40.39 / 6.7 / 7.7 / 99.0 / 99.0). What is the reason for this discrepancy?
-
How does retrieval complexity scale with: video length, memory size, number of generated chunks? Will the current design remain practical for larger videos (5 to 10 minutes)?
-
The method heavily depends on estimated cameras and depths. Can the authors provide robustness experiments with poses/depths artificially perturbed?
-
It is not clear if the performance gain against the baselines comes from the proposed architecture or from the used dataset. Would training baselines on WorldReplay reduce the performance gap?
5 comments
Response to Reviewer ei71 (Part 2 of 3)
3. Retrieval hyperparameters and selection heuristics
We added sensitivity experiments in Appendix C.2 and Figure 9. We vary semantic-anchor counts over 1, 2, 4, 8, and 16, and spatial keyframe counts over 1, 2, 3, 6, and 9, holding other settings fixed. The anchor sweep fixes three keyframes, and the keyframe sweep fixes eight anchors. AQ/IQ denote VBench aesthetic/imaging quality; FG PCK and Action Error assess multi-view synchronization.
| Varied parameter | Count | PSNR ↑ | AQ ↑ | IQ ↑ | FG PCK (%) ↑ | Action Error (°) ↓ |
|---|---|---|---|---|---|---|
| Anchors | 1 | 17.74 | 0.566 | 0.655 | 99.45 | 7.23 |
| Anchors | 2 | 17.88 | 0.568 | 0.656 | 99.37 | 7.10 |
| Anchors | 4 | 17.82 | 0.570 | 0.656 | 99.36 | 7.08 |
| Anchors | 8 | 17.73 | 0.572 | 0.660 | 99.46 | 7.29 |
| Anchors | 16 | 18.01 | 0.570 | 0.656 | 99.35 | 7.11 |
| Keyframes | 1 | 17.88 | 0.574 | 0.661 | 99.30 | 7.04 |
| Keyframes | 2 | 17.77 | 0.571 | 0.659 | 99.42 | 7.15 |
| Keyframes | 3 | 17.73 | 0.572 | 0.660 | 99.46 | 7.29 |
| Keyframes | 6 | 17.74 | 0.572 | 0.656 | 99.49 | 7.00 |
| Keyframes | 9 | 17.94 | 0.576 | 0.659 | 99.43 | 7.18 |
Revisit PSNR spans less than 0.3 dB in either sweep, with stable VBench scores. FG PCK ranges from 99.30% to 99.49% and Action Error from 7.00° to 7.29°. We invite the reviewer to examine the quality and synchronization curves in Figure 9 (right).
We use three spatial keyframes because spatiotemporal attention propagates their retrieved evidence across the chunk. More keyframes increase retrieval work and the context budget without consistent gains across metrics, so three is a practical balance. We use eight semantic anchors to reduce reliance on the VLM/LLM selecting a single informative frame. Although one anchor can preserve subject semantics, eight provide multiple identity references within the context budget; increasing to 16 gives no consistent gains across metrics. These choices balance cost and robustness, consistent with the weak sensitivity observed in the sweeps.
Greedy coverage selects complementary observations of target-visible regions at multiple times while limiting redundant views and patches under the context budget. To identify representative subject identity cues, a VLM captions sampled source frames, and an LLM selects semantic anchors from these descriptions. Our core contribution specifies what evidence to retrieve along time, space, and semantics. Other selection algorithms can supply this evidence. Our experiments assess the evidence design and count sensitivity; the necessity or superiority of specific selection algorithms is outside our scope.
4. Synchronization-number discrepancy
To reduce instability in human-pose estimation, we evaluate synchronization on 13 cases from our full evaluation set in which the dynamic subject is fully visible throughout both the input and target camera trajectories, meaning that all joints of the subject's skeleton remain visible along both trajectories.
The original Table 2 used 13 cases. When adding CameraAnything, we inadvertently configured Table 3's comparison list with 12 cases instead of 13. We apologize for this oversight and the resulting inconsistency.
The revised Tables 2 and 6 use the same 13-case set and report 99.33 / 40.83 / 6.5 / 7.4 / 98.6 / 98.8 for LongReplay. Appendix D documents the common eligibility rules. Figure 11's synchronization curves also use the corrected case list.
5. Retrieval complexity and 5–10-minute scalability
Let be the number of candidate memory frames, the number of patches per frame, the maximum number of selected frames, and the number of voxels used to compute coverage gain. With a fixed number of target keyframes, projection takes work. Assuming linear-time voxelization, we evaluate each candidate's coverage gain over at most voxels in each of greedy rounds, giving a retrieval cost estimate of . This is linear in candidate-memory size for fixed , , and . Increasing the selected-frame cap adds greedy scans while leaving the initial projection cost unchanged.
The DiT receives a fixed context budget of 32 latent-frame equivalents: temporal 12, spatial at most 12, and semantic 8. This bounds downstream attention input, while persistent memory grows with the source video and generated history. Across chunks, retrieval cost accumulates as . For fixed , , and , a candidate count that grows linearly with chunk index yields quadratic cumulative retrieval cost in the number of generated chunks.
These bounds indicate that the retrieval module can scale to 5–10-minute videos: its per-chunk context remains fixed while retrieval cost follows the candidate-memory growth described above.
Response to Reviewer ei71 (Part 3 of 3)
6. Robustness to perturbed poses and depths
These experiments are now included in Appendix C.1 and Figure 9. We perturb estimated reference-video geometry before constructing memory and correspondences. Depth receives independent pixelwise multiplicative Gaussian noise at 1%, 5%, 10%, and 20%. Cameras receive independent SE(3) perturbations at 10%, 100%, 200%, 500%, and 1000%. Camera percentages are measured relative to local inter-frame rotation and translation magnitudes: 100% means that the perturbation standard deviation equals the corresponding local motion magnitude. APE-t/R measure absolute camera translation/rotation error.
| Perturbation | PSNR ↑ | SSIM ↑ | APE-t ↓ | APE-R ↓ |
|---|---|---|---|---|
| None | 17.73 | 0.596 | 0.014 | 3.02 |
| Depth 1% | 17.79 | 0.596 | 0.015 | 3.00 |
| Depth 5% | 17.48 | 0.587 | 0.015 | 3.35 |
| Depth 10% | 17.03 | 0.579 | 0.014 | 3.37 |
| Depth 20% | 16.15 | 0.564 | 0.022 | 4.14 |
| Camera 10% | 17.92 | 0.596 | 0.014 | 3.02 |
| Camera 100% | 17.43 | 0.585 | 0.014 | 2.95 |
| Camera 200% | 17.54 | 0.588 | 0.015 | 3.37 |
| Camera 500% | 16.33 | 0.564 | 0.020 | 4.56 |
| Camera 1000% | 16.49 | 0.568 | 0.021 | 5.09 |
From an unperturbed PSNR of 17.73, 10% depth noise yields 17.03, a 0.70 dB decrease, and 200% camera noise yields 17.54, a 0.19 dB decrease. Greater degradation appears at 20% depth noise (16.15) and camera noise of 500% (16.33) or above. We invite the reviewer to examine Figure 9 (left), which plots performance across perturbation levels.
We also encourage the reviewer to watch ablation_geometry_robustness.mp4 in the supplementary ZIP. It shows the same example generated under different depth and camera noise levels, making the noise severity and its effects on the generated results easier to assess.
7. Architecture versus training-data advantage
We retrained Vista4D I2V* with the same Wan2.2-TI2V-5B backbone, WorldReplay data, and training protocol as LongReplay.
Results from Table 4 are summarized below; Appendix C.3 gives the full controlled comparison. APE-t/R measure absolute camera translation/rotation error; AQ/IQ are VBench aesthetic/imaging quality.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | APE-t ↓ | APE-R ↓ | AQ ↑ | IQ ↑ |
|---|---|---|---|---|---|---|---|
| Vista4D T2V | 16.85 | 0.570 | 0.448 | 0.125 | 58.13 | 0.543 | 0.656 |
| Vista4D I2V* | 16.91 | 0.595 | 0.388 | 0.013 | 3.08 | 0.561 | 0.678 |
| LongReplay | 17.73 | 0.596 | 0.387 | 0.014 | 3.02 | 0.572 | 0.660 |
Under the matched I2V setting, several aggregate metrics become comparable, while LongReplay retains higher revisit PSNR and aesthetic quality. Figure 10 and Figures 13–15 show clearer differences in dynamic synchronization, scene structure, and subject identity after re-entry. These examples highlight the difficulty of recovering changing subject states from static point-cloud conditioning. LongReplay instead retrieves relevant observations and preserves their source-time information through its correspondence design. With the backbone, WorldReplay data, and training protocol matched, these results support attributing the remaining advantages to our method design.
We encourage the reviewer to watch the side-by-side comparison videos in the supplementary ZIP, now including Vista4D I2V*, to assess these differences over time.
8. Presentation, terminology, and limitations
We have corrected “an Dynamic Scene” to “a Dynamic Scene” in Figure 1 and use Dynamic-Aware Correspondence, matching Section 3.4, consistently in Figure 2 and its caption. We will also simplify Figure 2's visual hierarchy to make the retrieval routes and correspondence module easier to follow in the final version.
The main-text Conclusion now includes limitations concerning quality degradation in regions never observed in the source video and non-interactive inference speed. Appendix E additionally discusses fine-grained identity drift and the limitations of source-only semantic anchors. The stated future directions include stronger backbones, faster sampling, and learnable semantic-anchor selection.
Response to Reviewer ei71 (Part 1 of 3)
Thank you for recognizing the motivation, dynamic-memory formulation, and value of WorldReplay, and for the concrete suggestions on related work and experimental clarity. We have expanded the revision with controlled baseline comparisons, ablations, robustness analyses, and protocol details. References below use the revision's numbering unless stated otherwise.
1. Missing related work and comparison with ReDirector
We have added ReDirector, RealCam, Reshoot-Anything, and Track2View to Section 2. ReDirector uses shared 3D RoPE and Rotary Camera Encoding for variable-length full-sequence retakes. RealCam interleaves source and target frames and distills a bidirectional teacher into a causal student for real-time streaming. Reshoot-Anything develops self-supervised reshooting from pseudo-multiview training, while Track2View conditions generation on paired 3D point tracks. LongReplay addresses persistent memory retrieval across autoregressive chunks, combining synchronized source evidence with spatial observations from multiple times and semantic anchors.
We did not compare with RealCam because its implementation has not been publicly released. Its finite KV window provides only short-term context and lacks persistent memory for long-range revisits, so we expect it to be less competitive in maintaining the long-term consistency required by our task.
We added a ReDirector comparison in Appendix C.4 and Table 5, with qualitative results in Figures 13–15. ReDirector jointly processes all source and target frames through DiT self-attention, so the attention matrix grows quadratically with the total sequence length. Its native implementation can process at most approximately 900 frames on an 80 GB GPU. We therefore evaluate both ReDirector and LongReplay on the same subset of videos shorter than 900 frames. APE-t/R measure absolute camera translation/rotation error.
ReDirector performs poorly on these long and complex target-camera trajectories, so we report the comparison in the appendix rather than the main text.
2. Distinction from HyDRA and UCM, and the method contribution
UCM. In UCM's “time-aware” positional encoding warping, the time coordinate is the target-frame index: a historical observation is encoded by (t,h^,w^) after projection into the target view. This warped encoding omits the original observation time t′. Observations from different source times can therefore receive the same correspondence coordinate when they project to the same target location. LongReplay additionally retains the original observation time in its correspondence coordinate, (Norm(t′),t,h^,w^). This matters because our retrieval intentionally supplies multiple observations of a region at different times: the model must distinguish their dynamic states when reconstructing the current event. Table 2(f) tests this variable directly. Removing t′ while retaining target coordinates and the same context allocation reduces revisit PSNR from 17.73 to 15.81 after retraining.
HyDRA. In HyDRA, “hybrid memory” refers to the dual goal of static-background consistency and dynamic-subject continuity. In LongReplay, “hybrid memory selection” refers to combining temporal, spatial, and semantic retrieval routes. The memory operations also differ: HyDRA uses affinity-based retrieval, while LongReplay retrieves the synchronized source event by timestamp, multiple observations of target-visible regions by geometry, and semantic anchors for subject identity. Our 4D correspondence retains observation time to distinguish dynamic states, while semantic correspondence links subject appearances across viewpoints. HyDRA evaluates consistency within fixed-length clips, including subject exit and re-entry; LongReplay targets scene consistency and source-event synchronization across autoregressive chunks.
HyDRA's HM-World contains short clips of subject exit–entry events built from a fixed set of 10 subject paths and 28 camera trajectory types. WorldReplay provides synchronized, one-minute-long multi-camera videos with randomized subject and camera trajectories, enabling more flexible motion combinations and scene exploration for paired supervision of long-video reshooting.
We thank the reviewer for suggesting that we make our innovations relative to UCM and HyDRA clearer. Section 2 now explicitly distinguishes our hybrid memory selection and dynamic-aware correspondence from these works.