acceptodds
Under review as a conference paper at ICLR 2027

LongReplay: Exporable Long Video Reshooting with Dynamic Memory

Abstract

Camera-controlled video reshooting synthesizes a novel-view video along a user-specified camera trajectory while remaining temporally synchronized with a source recording. Existing methods operate on short clips, where the source and the target occupy one attention window. We study long-video reshooting, in which the source may span minutes and the target is generated autoregressively in overlapping chunks. This setting requires synchronization under large camera baselines and large subject motion, together with long-term 3D consistency of the static scene and semantic consistency of dynamic subjects. Our key insight is to query a dynamic memory of the source video along time, space, and semantics. We propose LongReplay, which realizes this readout in two stages with dynamic-awareness. Memory selection retrieves a synchronized source chunk, multi-time spatial patches that cover the target field of view, and semantic anchors that store subject frames independently of the queried time and location. Dynamic-aware correspondence then identifies retrieved patches jointly by time and space with a 4D projection embedding, and matches tokens by semantics with TIPS descriptors. Supervising these queries requires paired long videos of the same dynamic scene under different cameras, so we construct WorldReplay, a Unreal Engine 5 pipeline that renders multiple camera trajectories in the same long take. Experiments show improved multi-view synchronization, including under large viewpoint changes, together with long-range geometric and identity consistency, relative to baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ei71OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles the problem of long-horizon, camera-controlled video reshooting. The authors propose to autoregressively generate, chunk by chunk, a novel-view video that follows a user-specified camera trajectory, lasts up to a few minutes, and remains synchronized with the source video. The approach formulates reshooting as a query of a dynamic memory along three axes: time (retrieving synchronized source chunks), space (spatial retrieval exploiting multi-time coverage of the same 3D point), and semantics (retrieving semantic anchors). The retrieved patches are concatenated with the noise tokens. Two correspondence embeddings are then added to the queries and keys (but not to the values): (1) a 4D correspondence embedding combining time and projected geometry, and (2) a semantic correspondence embedding based on TIPS descriptors.

The paper also introduces WorldReplay, a pipeline built with Unreal Engine 5 that produces one-minute, 8K synchronized multi-camera videos. The resulting dataset addresses the lack of long videos with synchronized multi-camera trajectories. Experiments show improvements over Vista4D, InSpatio-World, World-in-World, and CameraAnything in synchronization, camera control, and long-range consistency.

Critical strengths

The problem tackled in this paper is well motivated. The proposed solution, which decomposes long-horizon reshooting into temporal synchronization, spatial consistency, and semantic consistency, is intuitive and provides a useful conceptual framework for this problem.

The proposed approach simultaneously addresses several shortcomings of previous methods: large baselines, subjects that leave and re-enter the view, revisits, and accumulated camera drift. The paper also clearly explains how the approach resolves each of these issues.

If released, WorldReplay will be a useful dataset contribution for the community, especially if the authors also release the generation code so that it can be extended with larger scenarios and longer videos.

The paper includes meaningful ablations and shows strong improvements over the compared methods.

Critical weaknesses

Some close related works are missing:

  • Park et al. "Redirector: Creating any-length video retakes with rotary camera encoding." CVPR 2026.
  • Xu et al. "RealCam: Real-Time Novel-View Video Generation with Interactive Camera Control." arXiv preprint arXiv:2605.06051 (2026).
  • Paliwal et al. "Reshoot-anything: A self-supervised model for in-the-wild video reshooting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
  • Qiao et al. "Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks." arXiv preprint arXiv:2606.15534 (2026).

Retrieval design contains many heuristics: The framework relies on a number of heuristic design choices, for example: using only 3 keyframes for spatial retrieval, selecting 8 semantic anchors, the greedy coverage strategy for spatial selection, and the use of a VLM/LLM to choose the anchors. These choices appear somewhat arbitrary, the paper does not justify them, and there is no study of how sensitive the results are to changes in these parameters.

Novelty with respect to other memory-based world models is not sufficiently discussed. Several recent works are closely related:

  • Chen et al. "Out of sight but not out of mind: Hybrid memory for dynamic video world models." CoRR, abs/2603.25716, 2026b also propose a hybrid memory for dynamic scenes and introduce a synthetic dataset similar to the one presented here.
  • Xu et al. "UCM: Unified modeling of camera control and memory with time-aware positional encoding warping for world models." Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers. 2026 introduce a time-aware positional encoding warping that appears close to the 4D correspondence embedding proposed in this paper.

The related work section mostly lists prior papers without clearly explaining how the proposed method differs from them. For instance, how does the 4D embedding differ from UCM’s positional encoding warping? And how does the proposed memory differ from the hybrid memory of Chen et al.? In its current form, the paper reads more as an effective combination of existing ideas, applied to a new task and supported by a new dataset, than as a fundamentally new method. This is a legitimate and useful contribution, but the authors should state it clearly.

Presentation issues and typos:

  • Fig. 1: "Input Video of an Dynamic Scene"
  • Fig. 2 labels the module “Time-Aware Correspondence,” while the caption text calls it “Dynamic-Aware Correspondence.”
  • Fig. 2 is very dense and could be split into two figures
  • The limitations are not discussed in the main paper, but in the supplement materials.
Questions for authors
  • In the Appendix: Table 2’s full-model synchronization numbers (99.33 / 40.83 / 6.5 / 7.4 / 98.6 / 98.8) differ from Table 3’s LongReplay row (99.34 / 40.39 / 6.7 / 7.7 / 99.0 / 99.0). What is the reason for this discrepancy?

  • How does retrieval complexity scale with: video length, memory size, number of generated chunks? Will the current design remain practical for larger videos (5 to 10 minutes)?

  • The method heavily depends on estimated cameras and depths. Can the authors provide robustness experiments with poses/depths artificially perturbed?

  • It is not clear if the performance gain against the baselines comes from the proposed architecture or from the used dataset. Would training baselines on WorldReplay reduce the performance gap?

5 comments

Response to Reviewer ei71 (Part 1 of 3)

Thank you for recognizing the motivation, dynamic-memory formulation, and value of WorldReplay, and for the concrete suggestions on related work and experimental clarity. We have expanded the revision with controlled baseline comparisons, ablations, robustness analyses, and protocol details. References below use the revision's numbering unless stated otherwise.

1. Missing related work and comparison with ReDirector

We have added ReDirector, RealCam, Reshoot-Anything, and Track2View to Section 2. ReDirector uses shared 3D RoPE and Rotary Camera Encoding for variable-length full-sequence retakes. RealCam interleaves source and target frames and distills a bidirectional teacher into a causal student for real-time streaming. Reshoot-Anything develops self-supervised reshooting from pseudo-multiview training, while Track2View conditions generation on paired 3D point tracks. LongReplay addresses persistent memory retrieval across autoregressive chunks, combining synchronized source evidence with spatial observations from multiple times and semantic anchors.

We did not compare with RealCam because its implementation has not been publicly released. Its finite KV window provides only short-term context and lacks persistent memory for long-range revisits, so we expect it to be less competitive in maintaining the long-term consistency required by our task.

We added a ReDirector comparison in Appendix C.4 and Table 5, with qualitative results in Figures 13–15. ReDirector jointly processes all source and target frames through DiT self-attention, so the attention matrix grows quadratically with the total sequence length. Its native implementation can process at most approximately 900 frames on an 80 GB GPU. We therefore evaluate both ReDirector and LongReplay on the same subset of videos shorter than 900 frames. APE-t/R measure absolute camera translation/rotation error.

MethodPSNR ↑SSIM ↑LPIPS ↓APE-t ↓APE-R ↓
ReDirector13.970.5190.5770.16994.55
LongReplay18.550.6230.3630.0112.88

ReDirector performs poorly on these long and complex target-camera trajectories, so we report the comparison in the appendix rather than the main text.

2. Distinction from HyDRA and UCM, and the method contribution

UCM. In UCM's “time-aware” positional encoding warping, the time coordinate is the target-frame index: a historical observation is encoded by (t,h^,w^)(t,\hat{h},\hat{w}) after projection into the target view. This warped encoding omits the original observation time t′t'. Observations from different source times can therefore receive the same correspondence coordinate when they project to the same target location. LongReplay additionally retains the original observation time in its correspondence coordinate, (Norm(t′),t,h^,w^)(\mathrm{Norm}(t'),t,\hat{h},\hat{w}). This matters because our retrieval intentionally supplies multiple observations of a region at different times: the model must distinguish their dynamic states when reconstructing the current event. Table 2(f) tests this variable directly. Removing t′t' while retaining target coordinates and the same context allocation reduces revisit PSNR from 17.73 to 15.81 after retraining.

HyDRA. In HyDRA, “hybrid memory” refers to the dual goal of static-background consistency and dynamic-subject continuity. In LongReplay, “hybrid memory selection” refers to combining temporal, spatial, and semantic retrieval routes. The memory operations also differ: HyDRA uses affinity-based retrieval, while LongReplay retrieves the synchronized source event by timestamp, multiple observations of target-visible regions by geometry, and semantic anchors for subject identity. Our 4D correspondence retains observation time to distinguish dynamic states, while semantic correspondence links subject appearances across viewpoints. HyDRA evaluates consistency within fixed-length clips, including subject exit and re-entry; LongReplay targets scene consistency and source-event synchronization across autoregressive chunks.

HyDRA's HM-World contains short clips of subject exit–entry events built from a fixed set of 10 subject paths and 28 camera trajectory types. WorldReplay provides synchronized, one-minute-long multi-camera videos with randomized subject and camera trajectories, enabling more flexible motion combinations and scene exploration for paired supervision of long-video reshooting.

We thank the reviewer for suggesting that we make our innovations relative to UCM and HyDRA clearer. Section 2 now explicitly distinguishes our hybrid memory selection and dynamic-aware correspondence from these works.

Response to Reviewer ei71 (Part 2 of 3)

3. Retrieval hyperparameters and selection heuristics

We added sensitivity experiments in Appendix C.2 and Figure 9. We vary semantic-anchor counts over 1, 2, 4, 8, and 16, and spatial keyframe counts over 1, 2, 3, 6, and 9, holding other settings fixed. The anchor sweep fixes three keyframes, and the keyframe sweep fixes eight anchors. AQ/IQ denote VBench aesthetic/imaging quality; FG PCK and Action Error assess multi-view synchronization.

Varied parameterCountPSNR ↑AQ ↑IQ ↑FG PCK (%) ↑Action Error (°) ↓
Anchors117.740.5660.65599.457.23
Anchors217.880.5680.65699.377.10
Anchors417.820.5700.65699.367.08
Anchors817.730.5720.66099.467.29
Anchors1618.010.5700.65699.357.11
Keyframes117.880.5740.66199.307.04
Keyframes217.770.5710.65999.427.15
Keyframes317.730.5720.66099.467.29
Keyframes617.740.5720.65699.497.00
Keyframes917.940.5760.65999.437.18

Revisit PSNR spans less than 0.3 dB in either sweep, with stable VBench scores. FG PCK ranges from 99.30% to 99.49% and Action Error from 7.00° to 7.29°. We invite the reviewer to examine the quality and synchronization curves in Figure 9 (right).

We use three spatial keyframes because spatiotemporal attention propagates their retrieved evidence across the chunk. More keyframes increase retrieval work and the context budget without consistent gains across metrics, so three is a practical balance. We use eight semantic anchors to reduce reliance on the VLM/LLM selecting a single informative frame. Although one anchor can preserve subject semantics, eight provide multiple identity references within the context budget; increasing to 16 gives no consistent gains across metrics. These choices balance cost and robustness, consistent with the weak sensitivity observed in the sweeps.

Greedy coverage selects complementary observations of target-visible regions at multiple times while limiting redundant views and patches under the context budget. To identify representative subject identity cues, a VLM captions sampled source frames, and an LLM selects semantic anchors from these descriptions. Our core contribution specifies what evidence to retrieve along time, space, and semantics. Other selection algorithms can supply this evidence. Our experiments assess the evidence design and count sensitivity; the necessity or superiority of specific selection algorithms is outside our scope.

4. Synchronization-number discrepancy

To reduce instability in human-pose estimation, we evaluate synchronization on 13 cases from our full evaluation set in which the dynamic subject is fully visible throughout both the input and target camera trajectories, meaning that all joints of the subject's skeleton remain visible along both trajectories.

The original Table 2 used 13 cases. When adding CameraAnything, we inadvertently configured Table 3's comparison list with 12 cases instead of 13. We apologize for this oversight and the resulting inconsistency.

The revised Tables 2 and 6 use the same 13-case set and report 99.33 / 40.83 / 6.5 / 7.4 / 98.6 / 98.8 for LongReplay. Appendix D documents the common eligibility rules. Figure 11's synchronization curves also use the corrected case list.

5. Retrieval complexity and 5–10-minute scalability

Let NN be the number of candidate memory frames, P=H×WP=H\times W the number of patches per frame, BB the maximum number of selected frames, and VV the number of voxels used to compute coverage gain. With a fixed number of target keyframes, projection takes O(NP)O(NP) work. Assuming linear-time voxelization, we evaluate each candidate's coverage gain over at most VV voxels in each of BB greedy rounds, giving a retrieval cost estimate of O(NP+BNV)O(NP+BNV). This is linear in candidate-memory size for fixed PP, BB, and VV. Increasing the selected-frame cap adds greedy scans while leaving the initial projection cost unchanged.

The DiT receives a fixed context budget of 32 latent-frame equivalents: temporal 12, spatial at most 12, and semantic 8. This bounds downstream attention input, while persistent memory grows with the source video and generated history. Across chunks, retrieval cost accumulates as ∑iO(NiP+BNiV)\sum_i O(N_iP+BN_iV). For fixed PP, BB, and VV, a candidate count that grows linearly with chunk index yields quadratic cumulative retrieval cost in the number of generated chunks.

These bounds indicate that the retrieval module can scale to 5–10-minute videos: its per-chunk context remains fixed while retrieval cost follows the candidate-memory growth described above.

Response to Reviewer ei71 (Part 3 of 3)

6. Robustness to perturbed poses and depths

These experiments are now included in Appendix C.1 and Figure 9. We perturb estimated reference-video geometry before constructing memory and correspondences. Depth receives independent pixelwise multiplicative Gaussian noise at 1%, 5%, 10%, and 20%. Cameras receive independent SE(3) perturbations at 10%, 100%, 200%, 500%, and 1000%. Camera percentages are measured relative to local inter-frame rotation and translation magnitudes: 100% means that the perturbation standard deviation equals the corresponding local motion magnitude. APE-t/R measure absolute camera translation/rotation error.

PerturbationPSNR ↑SSIM ↑APE-t ↓APE-R ↓
None17.730.5960.0143.02
Depth 1%17.790.5960.0153.00
Depth 5%17.480.5870.0153.35
Depth 10%17.030.5790.0143.37
Depth 20%16.150.5640.0224.14
Camera 10%17.920.5960.0143.02
Camera 100%17.430.5850.0142.95
Camera 200%17.540.5880.0153.37
Camera 500%16.330.5640.0204.56
Camera 1000%16.490.5680.0215.09

From an unperturbed PSNR of 17.73, 10% depth noise yields 17.03, a 0.70 dB decrease, and 200% camera noise yields 17.54, a 0.19 dB decrease. Greater degradation appears at 20% depth noise (16.15) and camera noise of 500% (16.33) or above. We invite the reviewer to examine Figure 9 (left), which plots performance across perturbation levels.

We also encourage the reviewer to watch ablation_geometry_robustness.mp4 in the supplementary ZIP. It shows the same example generated under different depth and camera noise levels, making the noise severity and its effects on the generated results easier to assess.

7. Architecture versus training-data advantage

We retrained Vista4D I2V* with the same Wan2.2-TI2V-5B backbone, WorldReplay data, and training protocol as LongReplay.

Results from Table 4 are summarized below; Appendix C.3 gives the full controlled comparison. APE-t/R measure absolute camera translation/rotation error; AQ/IQ are VBench aesthetic/imaging quality.

MethodPSNR ↑SSIM ↑LPIPS ↓APE-t ↓APE-R ↓AQ ↑IQ ↑
Vista4D T2V16.850.5700.4480.12558.130.5430.656
Vista4D I2V*16.910.5950.3880.0133.080.5610.678
LongReplay17.730.5960.3870.0143.020.5720.660

Under the matched I2V setting, several aggregate metrics become comparable, while LongReplay retains higher revisit PSNR and aesthetic quality. Figure 10 and Figures 13–15 show clearer differences in dynamic synchronization, scene structure, and subject identity after re-entry. These examples highlight the difficulty of recovering changing subject states from static point-cloud conditioning. LongReplay instead retrieves relevant observations and preserves their source-time information through its correspondence design. With the backbone, WorldReplay data, and training protocol matched, these results support attributing the remaining advantages to our method design.

We encourage the reviewer to watch the side-by-side comparison videos in the supplementary ZIP, now including Vista4D I2V*, to assess these differences over time.

8. Presentation, terminology, and limitations

We have corrected “an Dynamic Scene” to “a Dynamic Scene” in Figure 1 and use Dynamic-Aware Correspondence, matching Section 3.4, consistently in Figure 2 and its caption. We will also simplify Figure 2's visual hierarchy to make the retrieval routes and correspondence module easier to follow in the final version.

The main-text Conclusion now includes limitations concerning quality degradation in regions never observed in the source video and non-interactive inference speed. Appendix E additionally discusses fine-grained identity drift and the limitations of source-only semantic anchors. The stated future directions include stronger backbones, faster sampling, and learnable semantic-anchor selection.

Reviewer_g4BgOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

LongReplay addresses offline, minute-scale novel-view reshooting of an existing dynamic video. It combines temporal retrieval, multi-time spatial retrieval, and semantic anchors with time-aware projection and TIPS-based correspondence. Autoregressive chunks are connected through overlapping frames and geometry-based camera calibration. WorldReplay provides synchronized multi-camera synthetic training videos. Experiments with full ablations evaluate camera control, revisits, perceptual quality, and cross-view foreground and human-pose consistency.

Critical strengths
  1. A well-motivated treatment of dynamic memory. Keeping the source observation time separate from its projected target location addresses a genuine ambiguity: observations of the same scene location can depict different states. The temporal, spatial, and semantic retrieval paths form a coherent response to synchronization and long-range consistency rather than an undifferentiated history buffer.

  2. Substantial supporting abation, evaluation and useful infrastructure. The main comparisons are complemented by quantitative correspondence ablations, camera-conditioned pose evaluation, threshold sensitivity, and consistency over video progress. WorldReplay supplies the paired long takes needed for this task. The disclosed training configuration and explicit discussion of non-interactive inference and identity drift are also helpful.

Critical weaknesses
  1. The independence and scope of evaluation need clearer documentation. The training corpus and synthetic test videos both use UE environments, but the scene/asset/animation/trajectory separation is not sufficiently explicit. Appendix C and Figure 9 also do not fully explain how the synchronization cases relate to the main 100-video evaluation, including the 13-versus-12-case difference. These are protocol uncertainties, not evidence of leakage or deliberate selection, but they affect the strength of the generalization claim.

  2. Attribution of the two central dynamic-memory choices is incomplete. Figure 5 isolates source-time information and multi-time coverage qualitatively. Table 2 provides valuable quantitative ablations, but replacing 4D projection with time-plus-Plücker conditioning does not isolate removal of source time alone. Separate quantitative results for those two variants would better establish their contributions. Existing evidence already supports the overall system; the gap concerns the precision of the component-level explanation.

Questions for authors
  1. Split clarification: How are the synthetic training and test sets separated by scene, character, animation, and camera trajectory? Please distinguish held-out scenes from new trajectories in familiar scenes.

  2. Evaluation scope: How do Tables 3–4 and Figure 9 map to the main 100 videos? Please explain case selection, eligible sample counts, the missing CameraAnything case, and whether summary comparisons use common cases. I acknowledge the shared-eligibility and threshold controls already reported.

  3. Attribution: Are quantitative results available for the source-time-only and multi-time-coverage ablations in Figure 5? Please clarify which ablations were retrained versus modified only at inference; an existing-metric comparison would suffice.

  4. Reproducibility and access: What will the release include for code, checkpoints, WorldReplay, and its generation pipeline? Please also specify the spatial-memory budget and end-to-end inference hardware, peak memory, and cost including geometric calibration.

6 comments

Response to Reviewer g4Bg (Part 1 of 2)

Thank you for your positive assessment of our work and thoughtful suggestions. We appreciate your recognition of our contributions and have revised the paper to clarify the points you raised. All section, table, and figure numbers below refer to the revision unless stated otherwise.

1. Synthetic training/test separation

All 50 synthetic test videos use UE5 scenes absent from the WorldReplay training set. The main evaluation contains these 50 synthetic videos and 50 in-the-wild videos.

Training and synthetic testing share the underlying character and animation asset libraries, but use different asset combinations and camera trajectories. Test trajectories comprise 40% random roaming, 30% 180-degree orbits, and 30% deliberate out-of-sight trajectories. Random roaming uses a generation strategy similar to that used in training, but with different trajectories. The other categories test large viewpoint changes and subjects leaving and re-entering the view. These details are now stated in Appendix A, under Training Dataset and Evaluation Dataset.

2. Synchronization subset, common eligibility, and the 13/12 discrepancy

The original Tables 3–4 and Figure 9 correspond to Tables 6–7 and Figure 11 in the revision. To reduce instability in human-pose estimation, we evaluate synchronization on a 13-case subset of the main 100-video evaluation in which the dynamic subject is fully visible throughout both the input and target camera trajectories, meaning that all joints of the subject's skeleton remain visible along both trajectories. Table 6 reports overall synchronization on this subset. Table 7 evaluates its samples with reference-defined viewpoint changes of at least 60∘60^\circ. Figure 11(a–c) reports scores over video progress, while Figure 11(d) varies the angular success threshold.

The numerical discrepancy came from a case-list configuration error. When we added CameraAnything to the baseline comparison, the evaluation case set was inadvertently configured to contain 12 cases instead of 13. The original Table 2 used 13 cases, whereas the original Table 3 used the 12-case list. This was an error in the comparison set, rather than a CameraAnything-specific generation or evaluation failure. We apologize for this oversight and the resulting inconsistency.

The revised Table 6 uses the same 13 cases for all methods, and its LongReplay row now matches Table 2: 99.33 / 40.83 / 6.5 / 7.4 / 98.6 / 98.8, in the order FG PCK, FG recall, orientation error, action error, 3D PCK, and joint success. The corresponding curves in Figure 11 have also been updated to use the corrected case list.

Response to Reviewer g4Bg (Part 2 of 2)

3. Source-time and multi-time-coverage ablations, including retraining status

We added quantitative results for both the source-time ablation (Table 2(f)) and the multi-time-coverage ablation (Table 3). The source-time variant is retrained, while the multi-time-coverage variant is modified only at inference, as documented in Appendix B.1–B.2.

All settings use the same context budget of 32 latent-frame equivalents, as requested by Reviewer YGnA.

SettingPSNR ↑SSIM ↑LPIPS ↓Training
Full model17.730.5960.387Retrained
Without source time t′t'15.810.5630.506Retrained
Without multi-time-coverage (FoV-based)16.340.5740.486Inference only

Removing source time retains the projected target coordinate (t,h,w)(t,h,w) and the same retrieval routes and token allocation, unlike the broader Plücker replacement in Table 2(a). Without t′t', the model cannot distinguish incompatible dynamic states in observations of the same scene location. Revisit PSNR drops by 1.92 dB, from 17.73 to 15.81.

Removing multi-time-coverage replaces spatial selection with UCM's FoV-based rule: one maximum-overlap context frame per target keyframe, using the original LongReplay weights. This provides insufficient evidence to distinguish mixed dynamic states, reducing revisit PSNR by 1.39 dB, from 17.73 to 16.34.

4. Release scope, spatial budget, and inference resources

We will publicly release LongReplay's code, model checkpoints, the WorldReplay dataset, and the UE5 generation pipeline.

Appendix A now reports inference on one NVIDIA A800 GPU, with peak memory usage of approximately 40–50 GB and an inference time of approximately 180 seconds per 49-frame chunk. DiT denoising takes approximately 150 seconds, and VAE encoding/decoding, VGGT-Omega calibration, and retrieval together take approximately 30 seconds.

Appendix A clarifies two spatial-retrieval budgets. The frame budget FS=18F_S=18 caps the number of contributing context frames and can be adjusted at inference without retraining. Greedy selection covers target-visible voxels with observations from multiple source times, and only the corresponding patches are injected into the model. A larger frame budget permits more selection iterations to cover more complex 3D scenes. In typical cases, 18 frames already suffice for multi-time coverage of all target-visible voxels, so we use this value to balance coverage and retrieval cost.

The patch budget 4HW4HW per target keyframe limits the injected context to at most four patches per target patch, where H×WH\times W is the patch-grid size. Across three target keyframes, this gives a spatial budget of 12 latent-frame equivalents per chunk. Actual token counts can be lower: two context patches from different times generally suffice for multi-time coverage.

Reviewer_g4BgOpenReview ↗

Follow-up: Reference Selection and Correction of Generated Memory

Thank you for the additional experiments and clarifications. I have two additional questions about practical use and possible extensions. Observations from existing examples or implementation experience would be sufficient; no additional experiments are requested.

  1. Selecting useful references from limited-view recordings. Appendix E already explains the limitations in regions never observed in the source. For long, mostly fixed-camera recordings of a person, my question instead concerns reference selection: based on existing experience, what makes an additional segment genuinely useful for preserving that individual during reshooting, rather than merely redundant? For example, do changes in visible surfaces, poses, or facial detail provide noticeably different benefits? Practical observations about useful versus unhelpful references would be valuable.

  2. Correcting generated memory when recorded evidence disagrees. Sections 3.2–3.3 describe storing source observations alongside generated history, and the response explains continual source anchoring. Is there an explicit preference for recorded observations when the two disagree, beyond the existing temporal retrieval route, or is reconciliation left to learned attention? As an open question, which change would seem most promising for allowing an earlier generated mistake to be corrected without sacrificing long-term consistency?

Reviewer_YGnAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes longreplay to address the long-video reshooting problem which generates a novel view video given a specified camera traj and synchronized with the source input video. The method combines temporal retrieval of the synchronized source chunk, spatial retrieval of observations from multiple times, and fixed semantic anchors. Retrieved tokens with source time aware geometric embeddings and TIPS based semantic correspondence distinguishs the changing object states and preserve identity across views. Generation proceeds autoregressively with first frame RGB and depth warp conditioning and camera error compensation. The authors also introduce worldreplay, a synthetic synchronized multi-camera dataset for training.

Critical strengths
  1. The demos look good in both quality and the preservation of the content after view point change.
  2. Combining multi-time retrieval with source time encoding is a sensible way to avoid mixing incompatible dynamic states.
  3. The contribution of the worldreplay dataset for synchronized long video generation.
  4. The method improves camera accuracy and revisit reconstruction over the evaluated baselines.
Critical weaknesses
  1. Vista4D uses a T2V checkpoint, although its official long video setup uses an I2V variant. This needs justification.
  2. Multi time retrieval and semantic anchors lack quantitative, equal context budget ablations.
  3. Robustness to estimated geometry and errors accumulated in generated history remains unclear.
Questions for authors
  1. Can you quantitatively evaluate source-time encoding, multi-time coverage, spatial retrieval, and semantic anchors while controlling the number of context tokens?
4 comments

Response to Reviewer YGnA (Part 1 of 2)

Thank you for recognizing the value of multi-time retrieval, source-time encoding, and WorldReplay. Table and figure numbers refer to the revision.

1. Vista4D baseline configuration

We used the publicly available Vista4D T2V checkpoint because its official long-video I2V weights were unavailable. We have now added a controlled comparison with Vista4D I2V*, retrained using the same Wan2.2-TI2V-5B backbone, WorldReplay training data, and training protocol as LongReplay. During training, ground-truth depth is used for point-cloud projection and rendering, and moving-actor masks filter dynamic pixels from the static point cloud.

Results from Table 4 are summarized below, with human-pose errors from Table 6. APE-t/R measure absolute camera translation/rotation error. AQ/IQ are VBench aesthetic/imaging quality.

MethodPSNR ↑SSIM ↑LPIPS ↓APE-t ↓APE-R ↓AQ ↑IQ ↑Orient. error (°) ↓Action error (°) ↓
Vista4D T2V16.850.5700.4480.12558.130.5430.65612.78.1
Vista4D I2V*16.910.5950.3880.0133.080.5610.6786.68.0
LongReplay17.730.5960.3870.0143.020.5720.6606.57.4

LongReplay improves revisit PSNR (17.73 vs. 16.91) and aesthetic quality (0.572 vs. 0.561), with comparable camera accuracy and lower human-pose errors.

In Figure 10, Vista4D I2V* shows an incorrect facing direction, inconsistent 3D structure, and changes in the subject's appearance after the subject leaves and re-enters the frame. LongReplay better preserves source dynamics, scene structure, and subject identity in these examples. We invite the reviewer to examine Figure 10 and the in-the-wild comparisons in Figures 13–15 to assess these qualitative differences alongside the aggregate metrics.

We also encourage the reviewer to watch the side-by-side comparison videos in the supplementary ZIP, which now include Vista4D I2V*, to assess synchronization and long-term consistency.

2. Quantitative attribution under a common context budget

The revision provides two complementary experiments: a source-time correspondence ablation in Table 2(f) and memory-route ablations in Table 3.

Table 2(f) removes only the source-time coordinate t′t' while retaining the projected target coordinate (t,h,w)(t,h,w) and the same retrieval routes and token allocation. Revisit PSNR falls from 17.73 to 15.81, a drop of 1.92 dB. This directly tests source-time information, separately from replacing projection with Plücker conditioning in Table 2(a). All Table 2 variants are retrained, as now stated in Appendix B.1.

Table 3 uses the same fixed context budget of 32 latent-frame equivalents, allocated across temporal, spatial, and semantic routes. Spatial retrieval supplies sparse patches up to a full-model limit equivalent to 12 latent frames and may use fewer tokens when coverage is already satisfied. When a route is removed or reduced, its budget is reassigned to the remaining routes. The revision clarifies that the spatial budget is an upper limit.

T/S/M denote temporal/spatial/semantic budgets in latent-frame equivalents. The source-time row is from Table 2(f). The other rows are from Table 3.

SettingMax. T+S+M budgetPSNR ↑SSIM ↑LPIPS ↓Training
Full model12+12+817.730.5960.387Retrained
Without source time t′t'12+12+815.810.5630.506Retrained
Without semantic anchors12+20+017.230.5860.418Retrained
FoV-based, no multi-time coverage12+3+1716.340.5740.486Inference only
Without spatial retrieval12+0+2015.660.5660.494Retrained

Removing spatial retrieval reduces PSNR by 2.07 dB, and removing semantic anchors reduces it by 0.50 dB, even after reallocating their budgets. The FoV-based variant also lacks sufficient evidence to distinguish mixed dynamic states, reducing PSNR by 1.39 dB.

Response to Reviewer YGnA (Part 2 of 2)

3. Geometry robustness and accumulated history errors

We have added geometry perturbation experiments in Appendix C.1 and Figure 9. Before constructing retrieval memory and correspondences, we perturb estimated reference-video depth with independent pixelwise multiplicative Gaussian noise at 1%, 5%, 10%, and 20%. Camera poses receive independent SE(3) perturbations at 10%, 100%, 200%, 500%, and 1000%. For cameras, 100% means that the perturbation standard deviation equals the local inter-frame rotation or translation magnitude. Appendix C.1 specifies the full protocol. APE-t/R measure absolute camera translation/rotation error.

PerturbationPSNR ↑SSIM ↑APE-t ↓APE-R ↓
None17.730.5960.0143.02
Depth 1%17.790.5960.0153.00
Depth 5%17.480.5870.0153.35
Depth 10%17.030.5790.0143.37
Depth 20%16.150.5640.0224.14
Camera 10%17.920.5960.0143.02
Camera 100%17.430.5850.0142.95
Camera 200%17.540.5880.0153.37
Camera 500%16.330.5640.0204.56
Camera 1000%16.490.5680.0215.09

Revisit PSNR is 17.73 without perturbation, 17.03 with 10% depth noise, and 17.54 with 200% camera noise. These correspond to drops of 0.70 and 0.19 dB. Larger perturbations produce clearer degradation: PSNR reaches 16.15 at 20% depth noise and 16.33 at 500% camera noise. We invite the reviewer to examine Figure 9 (left) in the revision, which plots performance across depth and camera perturbation levels.

We also encourage the reviewer to watch ablation_geometry_robustness.mp4 in the supplementary ZIP. It shows the same example generated under different depth and camera noise levels, making the noise severity and its effects on the generated results easier to assess.

The source video and its estimated geometry remain fixed references throughout the rollout. Synchronized source observations anchor each chunk's dynamics and appearance. Generated chunks are geometrically calibrated against previously calibrated context and aligned to the world frame before entering memory. Error-aware camera blending compensates for the overlapping frame's residual camera error in the next chunk's warp condition. This continual source anchoring suppresses drift across chunks, so perturbing the source geometry tests the robustness of the geometric reference used to control accumulated history error.

Visual drift remains a limitation, particularly for fine-grained identity cues and regions never observed in the source video. A stronger backbone could alleviate these failures, while learnable semantic-anchor selection could improve consistency for newly observed parts of dynamic subjects. We leave both directions for future work.

Dear Reviewer,

we sincerely appreciate your positive voting and valuable suggestions. Since there are only a few days left in the discussion period, to ensure that we have sufficient time to provide additional materials later, we kindly ask if our responses have addressed your concerns.

We also welcome any further questions you may have and will actively respond to them.

Best regards,

LongReplay Authors

I appreciate the responces from the authors and I will maintain my postive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Description

Summary of Revisions

We thank the reviewers for their constructive feedback. We have revised the manuscript and supplementary material as summarized below. All section, table, and figure numbers refer to the revised manuscript. Reviewer IDs in brackets indicate the review comments addressed by each item.

  1. Added a controlled Vista4D I2V comparison. [YGnA, ei71] We retrained Vista4D I2V with the same Wan2.2-TI2V-5B backbone, WorldReplay data, and training protocol as LongReplay. We report quantitative results and qualitative comparisons of dynamic synchronization, scene consistency, and subject identity (Appendix C.3, Tables 4 and 6, Figure 10, and Figures 13–15). The side-by-side comparison videos in the supplementary ZIP now also include Vista4D I2V*.

  2. Added comparisons with ReDirector. [ei71] We report quantitative results on the same compatible test subset for both methods, explain ReDirector's full-sequence processing constraints, and include qualitative comparisons on in-the-wild videos (Appendix C.4, Table 5, and Figures 13–15).

  3. Added quantitative component ablations. [YGnA, g4Bg] We isolate removal of the source-time coordinate and evaluate multi-time coverage, spatial retrieval, and semantic anchors under a fixed context budget. We specify budget redistribution and distinguish retrained variants from the inference-only FoV-based variant (Appendix B.1–B.2 and Tables 2–3).

  4. Added geometry robustness experiments. [YGnA, ei71] We evaluate generation under controlled depth and camera-pose perturbations, document the noise definitions, and explain how source anchoring relates these experiments to accumulated geometry errors. The supplementary ZIP includes ablation_geometry_robustness.mp4, showing warped observations and generated results for the same example under different noise levels (Appendix C.1 and Figure 9, left).

  5. Added hyperparameter sensitivity analyses. [ei71] We vary semantic-anchor counts over 1, 2, 4, 8, and 16, and spatial retrieval keyframe counts over 1, 2, 3, 6, and 9. We report the resulting quality and synchronization curves, and explain the default choices of eight anchors and three keyframes (Appendix C.2 and Figure 9, right).

  6. Expanded related work and clarified methodological distinctions. [ei71] We add ReDirector, RealCam, Reshoot-Anything, and Track2View, and explicitly distinguish our source-time-preserving correspondence from UCM's positional encoding warping and our temporal, spatial, and semantic retrieval from HyDRA's hybrid memory (Section 2).

  7. Clarified the dataset and evaluation split. [g4Bg] We correct the dataset-size description and specify that all 50 synthetic test videos use unseen UE5 scenes, while character and animation libraries are shared with training. We also document the different asset combinations and camera trajectories, including the test trajectory proportions (Appendix A).

  8. Corrected and clarified synchronization evaluation. [ei71, g4Bg] We explain the synchronization subset selection and correct the 13-versus-12 case-list discrepancy. The comparison and ablation tables now use the same 13-case set, and the corresponding synchronization curves have been updated (Appendix D, Tables 2 and 6, and Figure 11).

  9. Expanded implementation and resource details. [g4Bg] We describe the spatial retrieval procedure, including multi-time coverage, patch selection, and budget control, and report inference hardware, peak memory, and end-to-end runtime (Appendix A).

  10. Improved terminology and limitation reporting. [ei71] We correct the Figure 1 typo, align Figure 2's terminology with Dynamic-Aware Correspondence, and add limitations and future directions to the main-text Conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.