- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReCoG, a framework for subject-aware camera-controllable video background generation that aims to improve camera control accuracy and reference consistency while preserving the flexibility of Reference-to-Video (R2V) generation. The authors identify the lack of explicit reference-to-video correspondence as a key limitation of existing R2V methods and observe that such correspondence emerges internally within the self-attention of diffusion Transformers during early denoising. Based on this observation, ReCoG introduces Anchor Block Selection (ABS) to identify informative correspondences and a Sparse Anchoring Strategy (SAS) with Noise-Resilient Matching (NRM) to refine noisy token-level matches into reliable sparse geometric anchors. These anchors are propagated along the target camera trajectory through depth warping and injected into subsequent Transformer blocks via Sparse Anchor Injection (SAI). Additionally, Anchor Distribution Alignment (ADA) constructs ground-truth-derived sparse anchors to stabilize training. Experiments on the synthetic MultiCamVideo and real-world DL3DV datasets demonstrate improvements in camera control accuracy, reference consistency, and video quality over the evaluated baselines. Ablation studies further investigate the effectiveness of the proposed components.
Critical strengths
- The paper presents a clear motivation by highlighting the trade-off between I2V's deterministic correspondences but limited flexibility and R2V's greater adaptability but lack of explicit geometric constraints, justifying the introduction of explicit correspondences and geometric guidance.
- The proposed method and modules are well aligned with the paper's motivation, demonstrating clear logical consistency. To address the lack of explicit geometric constraints in R2V, ABS, SAS, and NRM extract correspondences and construct sparse anchors, while SAI introduces explicit geometric guidance and ADA mitigates non-stationary conditioning during training. Together, these modules form a coherent framework for improving correspondence reliability and camera control.
- The overall framework diagram is clear and visually appealing, providing an intuitive illustration of the proposed method. Additionally, the extensive qualitative visualizations enhance the paper's readability and strengthen the persuasiveness of its experimental results.
Critical weaknesses
- Although ReCoG outperforms the evaluated baselines, comparisons are limited to VACE+FT and VACE+UCPE, which may not faithfully represent ActAnywhere and DynaScene. Moreover, the comparison with GEN3C is qualitative only. These limitations weaken the evidence for ReCoG's competitiveness against existing state-of-the-art methods.
- Although ReCoG relies on endogenous correspondences emerging from DiT self-attention, this assumption is mainly supported by Figure 2 and final camera control metrics. Equations (4) and (5) measure attention responses rather than geometric correctness. Additional quantitative evaluations of correspondence accuracy, reprojection error, and correspondence quality across denoising stages are needed to validate this core assumption.
- Although ADA aims to mitigate non-stationary conditioning, the paper does not verify whether training and inference anchor distributions are aligned. Training anchors use random GT sampling and masking, whereas inference anchors rely on SAS and NRM. Table 2 demonstrates performance gains but does not establish distribution alignment. Further analyses of anchor distributions, training convergence, and masking strategies are needed.
- Although ReCoG emphasizes R2V's flexibility in adapting to different reference viewpoints and scene layouts, NRM in Equation (6) requires the matched reference region to share the same region label as the spatially corresponding location of the current video token in the reference image. This constraint may implicitly assume strong spatial layout consistency, potentially filtering out valid correspondences under large viewpoint changes or layout shifts. While Appendix E reports failures under extreme scale and perspective differences, it remains unclear whether these failures are caused by NRM. Additional analyses of correspondence retention rates and generation performance under varying viewpoint and layout changes are needed to validate the rationale and applicability of this constraint.
Questions for authors
see weakness
We thank the reviewer for the constructive feedback. We have added further experiments and clarifications in the revised manuscript.
W1: Competitiveness against existing methods
To better establish the competitiveness of ReCoG, we expanded our evaluation with GEN3C and Bernini, covering camera-controlled I2V generation and video background replacement, respectively. Quantitative and qualitative results are provided in Appendix E (Table 8 and Fig. 16).
Compared with GEN3C, ReCoG reduces rotation error by 14.9%, FVD by 23.9%, and LPIPS by 4.1%. While GEN3C achieves lower translation error and both baselines obtain higher reference similarity on certain metrics, qualitative comparisons reveal that faithfully preserving reference content does not necessarily ensure spatial compatibility with the foreground. ReCoG instead balances camera adherence, reference consistency, and subject-aware adaptation. Notably, Bernini does not support explicit target camera trajectory conditioning.
These comparisons complement our shared-backbone evaluations (Table 1), where we adopt UCPE, a state-of-the-art implicit camera-pose conditioning method, as a strong baseline. Under the same pretrained DiT backbone, ReCoG reduces translation and rotation errors. This controlled comparison highlights the advantages of our explicit correspondence-derived geometric guidance over advanced implicit camera conditioning, while minimizing architectural confounding factors.
We believe these complementary evaluations provide strong evidence of ReCoG's competitiveness against existing methods and the effectiveness of our design under controlled settings.
W2: Geometric validity of endogenous correspondence
We added a geometric correspondence accuracy analysis in Appendix D (Geometric correspondence accuracy analysis, Fig. 13). Since each video token represents an image patch, a match is considered valid if at least one GT geometric correspondence exists between the two associated patches. The results show that meaningful geometric correspondences emerge during early denoising, with notable variations across Transformer blocks. NRM further improves the accuracy and quality of the extracted correspondences.
W3: Clarification of Anchor Distribution Alignment
We apologize for the ambiguity in the term "distribution alignment." Our intention is to align the spatiotemporal patterns of anchor conditioning, rather than perform explicit statistical distribution matching. We clarify the design rationale of ADA below.
First, the w/o ADA ablation in Table 2 shows that using non-stationary conditioning during training degrades performance, motivating a more stable training strategy. Since the inference-time anchor video is constructed by depth-warping a single sparse anchor image, we adopt a similar conditioning structure during training. Specifically, ADA randomly samples a GT frame, applies stochastic masking, and then performs depth warping along the camera trajectory. By learning to synthesize complete videos from such sparse, geometrically consistent guidance, the model can utilize similarly structured anchors extracted by SAS and NRM at inference.
We further investigate alternative masking strategies and ratios in Appendix D (Analysis of masking strategies, Fig. 15 and Table 7). Randomly masking all GT video frames introduces temporal redundancy, allowing the model to exploit neighboring observations and potentially reproduce masked regions as artifacts. In contrast, our single-frame masking produces temporally coherent sparse guidance. Regarding masking ratios, 0.50 favors video quality and reference consistency, whereas 0.90 achieves the best camera-control accuracy. Increasing the ratio to 0.95 further weakens geometric guidance and degrades camera control. We therefore adopt 0.90 as the default.
W4. Concern about failure cases under extreme scale and perspective differences.
We would like to clarify that the observed failure cases primarily reflect the scene adaptation limitations of the underlying R2V backbone, rather than NRM. As shown in Appendix F (Fig. 17), VACE+FT exhibits similar failures without NRM under extreme viewpoint and scale mismatches.
We further evaluate correspondence retention and generation performance across viewpoint differences in Appendix F (Fig. 18). The results show that NRM consistently improves camera accuracy and reference consistency below 40°, while maintaining advantages in most metrics beyond 40°, except translation error. We acknowledge that, although ReCoG demonstrates greater flexibility in adapting to reference viewpoint changes than existing baselines in the evaluated scenarios, extreme viewpoint differences remain challenging. We added a discussion of this limitation in Appendix F.