acceptodds
Under review as a conference paper at ICLR 2027

ReCoG: Reifying Endogenous Correspondence for Subject-Aware Camera-Controllable Video Background Generation

Abstract

Subject-aware camera-controllable video background generation aims to synthesize a background video from a reference image that flexibly adapts to the subject while following a prescribed camera trajectory. Existing Reference-to-Video (R2V) methods provide flexible scene conditioning, but lack explicit reference-to-video correspondence, forcing the model to implicitly infer reference geometry and often resulting in inaccurate camera control and reference inconsistency. We reveal that such correspondence, although absent from the R2V formulation, emerges internally during diffusion. Building on this finding, we propose ReCoG, which reifies endogenous reference-to-video correspondence into explicit geometric guidance. ReCoG first locates informative correspondence within the Transformer, then exploits region-level coherence to refine noisy early-denoising matches, and materializes the reliable correspondences into sparse anchors that are warped along the target camera trajectory. Ground-truth-derived anchors that approximate the inference-time sparse-anchor distribution further stabilize training. Across synthetic and real-world evaluations, ReCoG achieves the best performance in both reference consistency and camera control accuracy among the compared methods. These results demonstrate that endogenous correspondence provides an effective route to explicit geometric control while retaining the flexibility of R2V generation.

Primary area:
generative models
Keywords:
Video Generation, Reference-to-video, Camera Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oJckOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCoG, a framework for subject-aware camera-controllable video background generation that aims to improve camera control accuracy and reference consistency while preserving the flexibility of Reference-to-Video (R2V) generation. The authors identify the lack of explicit reference-to-video correspondence as a key limitation of existing R2V methods and observe that such correspondence emerges internally within the self-attention of diffusion Transformers during early denoising. Based on this observation, ReCoG introduces Anchor Block Selection (ABS) to identify informative correspondences and a Sparse Anchoring Strategy (SAS) with Noise-Resilient Matching (NRM) to refine noisy token-level matches into reliable sparse geometric anchors. These anchors are propagated along the target camera trajectory through depth warping and injected into subsequent Transformer blocks via Sparse Anchor Injection (SAI). Additionally, Anchor Distribution Alignment (ADA) constructs ground-truth-derived sparse anchors to stabilize training. Experiments on the synthetic MultiCamVideo and real-world DL3DV datasets demonstrate improvements in camera control accuracy, reference consistency, and video quality over the evaluated baselines. Ablation studies further investigate the effectiveness of the proposed components.

Critical strengths
  1. The paper presents a clear motivation by highlighting the trade-off between I2V's deterministic correspondences but limited flexibility and R2V's greater adaptability but lack of explicit geometric constraints, justifying the introduction of explicit correspondences and geometric guidance.
  2. The proposed method and modules are well aligned with the paper's motivation, demonstrating clear logical consistency. To address the lack of explicit geometric constraints in R2V, ABS, SAS, and NRM extract correspondences and construct sparse anchors, while SAI introduces explicit geometric guidance and ADA mitigates non-stationary conditioning during training. Together, these modules form a coherent framework for improving correspondence reliability and camera control.
  3. The overall framework diagram is clear and visually appealing, providing an intuitive illustration of the proposed method. Additionally, the extensive qualitative visualizations enhance the paper's readability and strengthen the persuasiveness of its experimental results.
Critical weaknesses
  1. Although ReCoG outperforms the evaluated baselines, comparisons are limited to VACE+FT and VACE+UCPE, which may not faithfully represent ActAnywhere and DynaScene. Moreover, the comparison with GEN3C is qualitative only. These limitations weaken the evidence for ReCoG's competitiveness against existing state-of-the-art methods.
  2. Although ReCoG relies on endogenous correspondences emerging from DiT self-attention, this assumption is mainly supported by Figure 2 and final camera control metrics. Equations (4) and (5) measure attention responses rather than geometric correctness. Additional quantitative evaluations of correspondence accuracy, reprojection error, and correspondence quality across denoising stages are needed to validate this core assumption.
  3. Although ADA aims to mitigate non-stationary conditioning, the paper does not verify whether training and inference anchor distributions are aligned. Training anchors use random GT sampling and masking, whereas inference anchors rely on SAS and NRM. Table 2 demonstrates performance gains but does not establish distribution alignment. Further analyses of anchor distributions, training convergence, and masking strategies are needed.
  4. Although ReCoG emphasizes R2V's flexibility in adapting to different reference viewpoints and scene layouts, NRM in Equation (6) requires the matched reference region to share the same region label as the spatially corresponding location of the current video token in the reference image. This constraint may implicitly assume strong spatial layout consistency, potentially filtering out valid correspondences under large viewpoint changes or layout shifts. While Appendix E reports failures under extreme scale and perspective differences, it remains unclear whether these failures are caused by NRM. Additional analyses of correspondence retention rates and generation performance under varying viewpoint and layout changes are needed to validate the rationale and applicability of this constraint.
Questions for authors

see weakness

1 comment

We thank the reviewer for the constructive feedback. We have added further experiments and clarifications in the revised manuscript.


W1: Competitiveness against existing methods

To better establish the competitiveness of ReCoG, we expanded our evaluation with GEN3C and Bernini, covering camera-controlled I2V generation and video background replacement, respectively. Quantitative and qualitative results are provided in Appendix E (Table 8 and Fig. 16).

Compared with GEN3C, ReCoG reduces rotation error by 14.9%, FVD by 23.9%, and LPIPS by 4.1%. While GEN3C achieves lower translation error and both baselines obtain higher reference similarity on certain metrics, qualitative comparisons reveal that faithfully preserving reference content does not necessarily ensure spatial compatibility with the foreground. ReCoG instead balances camera adherence, reference consistency, and subject-aware adaptation. Notably, Bernini does not support explicit target camera trajectory conditioning.

These comparisons complement our shared-backbone evaluations (Table 1), where we adopt UCPE, a state-of-the-art implicit camera-pose conditioning method, as a strong baseline. Under the same pretrained DiT backbone, ReCoG reduces translation and rotation errors. This controlled comparison highlights the advantages of our explicit correspondence-derived geometric guidance over advanced implicit camera conditioning, while minimizing architectural confounding factors.

We believe these complementary evaluations provide strong evidence of ReCoG's competitiveness against existing methods and the effectiveness of our design under controlled settings.


W2: Geometric validity of endogenous correspondence

We added a geometric correspondence accuracy analysis in Appendix D (Geometric correspondence accuracy analysis, Fig. 13). Since each video token represents an image patch, a match is considered valid if at least one GT geometric correspondence exists between the two associated patches. The results show that meaningful geometric correspondences emerge during early denoising, with notable variations across Transformer blocks. NRM further improves the accuracy and quality of the extracted correspondences.


W3: Clarification of Anchor Distribution Alignment

We apologize for the ambiguity in the term "distribution alignment." Our intention is to align the spatiotemporal patterns of anchor conditioning, rather than perform explicit statistical distribution matching. We clarify the design rationale of ADA below.

First, the w/o ADA ablation in Table 2 shows that using non-stationary conditioning during training degrades performance, motivating a more stable training strategy. Since the inference-time anchor video is constructed by depth-warping a single sparse anchor image, we adopt a similar conditioning structure during training. Specifically, ADA randomly samples a GT frame, applies stochastic masking, and then performs depth warping along the camera trajectory. By learning to synthesize complete videos from such sparse, geometrically consistent guidance, the model can utilize similarly structured anchors extracted by SAS and NRM at inference.

We further investigate alternative masking strategies and ratios in Appendix D (Analysis of masking strategies, Fig. 15 and Table 7). Randomly masking all GT video frames introduces temporal redundancy, allowing the model to exploit neighboring observations and potentially reproduce masked regions as artifacts. In contrast, our single-frame masking produces temporally coherent sparse guidance. Regarding masking ratios, 0.50 favors video quality and reference consistency, whereas 0.90 achieves the best camera-control accuracy. Increasing the ratio to 0.95 further weakens geometric guidance and degrades camera control. We therefore adopt 0.90 as the default.


W4. Concern about failure cases under extreme scale and perspective differences.

We would like to clarify that the observed failure cases primarily reflect the scene adaptation limitations of the underlying R2V backbone, rather than NRM. As shown in Appendix F (Fig. 17), VACE+FT exhibits similar failures without NRM under extreme viewpoint and scale mismatches.

We further evaluate correspondence retention and generation performance across viewpoint differences in Appendix F (Fig. 18). The results show that NRM consistently improves camera accuracy and reference consistency below 40°, while maintaining advantages in most metrics beyond 40°, except translation error. We acknowledge that, although ReCoG demonstrates greater flexibility in adapting to reference viewpoint changes than existing baselines in the evaluated scenarios, extreme viewpoint differences remain challenging. We added a discussion of this limitation in Appendix F.

Reviewer_BhcTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on background generation conditioned on a foreground-subject video, a background reference image, and a target camera trajectory. It extract video-to-reference correspondences from the attention of video generator and convert them into geometric guidance in generation process. And it filters noisy matches using regions obtained from reference self-attention, and constructs a sparse image and depth anchor. Then, this anchor is reprojected along the target camera trajectory and injected into subsequent blocks. During training, sparse anchors are constructed from ground-truth frames with random masking. Experiments on MultiCamVideo and DL3DV datasets show improvements over the two VACE-based baselines in camera accuracy, video quality and reference consistency.

Critical strengths
  1. The idea of this paper is intuitive, recovering correspondence cues already present inside the generator and take them into geometric guidance for camera control.
  2. The sparse anchors offer a sensible balance between constraining scene motion and leaving space for the background to adapt to the foreground subject.
  3. The module for filtering noisy matches at the region level is a reasonable choice, since coarse associations may be more reliable than precise token matches during early generation.
Critical weaknesses
  1. The showing results are promising, though the contribution of correspondence recovery could be better isolated. And removing NRM also changes anchor density, so a comparison with the same number of retained matches would strengthen the effect for the filtering strategy.
Questions for authors
  1. In the early denoising process, region-level filtering is a sensible way to handle noisy early matches. It would be better to compare NRM with random or attention-confidence filtering while keeping the number of retained matches fixed.

  2. The method for caching the anchor avoids repeated extraction and keeps the guidance fixed during denoising. It would be interesting to compare whether refreshing it once later improves the result or have a bad effect on scene stability.

3 comments

We thank the reviewer for the constructive suggestions. We have conducted two additional experiments and included the results in the revised manuscript.


W1&Q1: Fixed-Match-Count Comparison

To isolate the effect of NRM from anchor density, we compare it with random sampling (Random-K) and attention-confidence filtering (Top-K), keeping the number of retained matches identical for each sample.

MethodTransErr↓RotErr↓FID↓FVD↓LPIPS↓SSIM↑SC↑BC↑TF↑MS↑DS-Max↑DS-Avg↑
Random-K0.14033.6599212.821637.980.54920.39330.93650.91980.96990.99010.67650.6304
Top-K0.15313.7891212.131621.030.54920.39200.93650.92000.96990.99010.67560.6308
NRM (Ours)0.12612.9499169.481292.380.48840.43890.93720.92080.97060.99170.73030.6802

NRM consistently outperforms both alternatives across all metrics, demonstrating that region-level matching provides benefits beyond anchor sparsification. Additional qualitative comparisons of the extracted anchors and generated videos are provided in Appendix D (Fixed-match-count comparison, Fig. 11).


Q2: Anchor Refreshing

We evaluate a single anchor refresh at step 5 (early) or 10 (middle) of the 20-step denoising process.

Refresh StepTransErr↓RotErr↓FID↓FVD↓LPIPS↓SSIM↑SC↑BC↑TF↑MS↑DS-Max↑DS-Avg↑
0 (Ours)0.12612.9499169.481292.380.48840.43890.93720.92080.97060.99170.73030.6802
50.12853.0198171.381324.500.48980.43690.93670.92030.97100.99180.71890.6676
100.12513.3015169.741302.530.48510.43980.93720.92080.97080.99170.72330.6727

Neither refresh setting provides consistent improvements. We therefore retain the default strategy of extracting the anchor only once, avoiding additional computation. Appendix D (Anchor refreshing, Fig. 12) further presents qualitative comparisons of the anchor images and generated videos under different refresh strategies.

Thank you for your response. Most of my concerns have been addressed, and I will maintain my positive rating.

Thank you for your positive feedback and for acknowledging our efforts to address your concerns. We sincerely appreciate your constructive comments and support.

Reviewer_wy88OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ReCoG, for video generation from a given foreground video, a background reference image, and camera trajectory. The paper identified the anchor DiT block which serves as the key component to extract the correspondence between the generated video and reference image. Based on this, the paper proposed the SAS, ADA, and SAI mechanisms to achieve the background video generation.

Critical strengths
  1. The use of internal model information is interesting.
  2. The paper is generally well-structured and easy to follow.
  3. The experiments show siginificant performance improvement compared to the baseline.
Critical weaknesses
  1. The paper is based on the assumption that, the anchor block significantly affect the extraction of correspondence of the generated video and reference image. However, the paper did not provide direct evidence to support such claim. I understand the performance improvement on the downstream task circumstantially support this assumption, but it would not fully justify this claim.
  2. Insufficient experiment: The quantitative results in Tab. 1 only involve two baselines with the same backbone, which is not sufficient for a universal conclusion. The proposed method is only tested on a single backbone fundation model, it is unknown if the "anchor block" claims still hold for other backbone models.
  3. The anchor block selection is empirically selected by investing the attention scores and visual check, lacking direct assessment such as correspondence accuracy.
Questions for authors

The statement "NRM provides marginal improvements in Temporal Flickering (TF) and Motion Smoothness (MS)" in Line 431 is unclear, does the NRM increase or decrease the TF and MS?

3 comments

We thank the reviewer for the constructive feedback. We have added further experiments and clarifications in the revised manuscript.


W1: Direct evidence for geometric correspondence

We added a geometric correspondence accuracy analysis in Appendix D (Geometric correspondence accuracy analysis, Fig. 13). Since each video token represents an image patch, a match is considered valid if at least one GT geometric correspondence exists between the two associated patches. The results show that the selected Anchor Block captures meaningful geometric correspondences, with substantial improvements after NRM.

W2: Shared backbone and generalizability

Using the same pretrained backbone is an intentional choice to ensure controlled comparisons, isolating the contribution of ReCoG from differences in model architecture. We further conduct a cross-backbone analysis on VACE-LTX in Appendix D (Cross-backbone analysis, Fig. 14), which differs from VACE-Wan in its architecture and attention implementation. The results reveal similar patterns of endogenous correspondence and demonstrate preliminary background video generation capability, providing initial evidence of cross-backbone applicability. A more comprehensive evaluation and optimization of ReCoG on this backbone are left for future work.

W3: Concern about the anchor block selection by attention scores and visual check

We added correspondence accuracy analysis in Appendix D (Geometric correspondence accuracy analysis, Fig. 13) to complement the original quantitative ablations in Table 2 (Sec. 4.3). Although Block 18 achieves higher correspondence accuracy than Block 15, it leaves fewer subsequent blocks for geometric guidance injection. Table 2 shows that Block 15 achieves better camera-control accuracy, supporting the importance of balancing correspondence quality with subsequent guidance capacity.

Q1: Clarification of TF and MS

We apologize for the typo. The correct statement is:"While removing NRM provides marginal improvements in Temporal Flickering (TF) and Motion Smoothness (MS), these gains come at the cost of substantial degradation in fine-grained details."

Thank you for the reply. I have some more following questions.

  1. In Sec. D (Geometric correspondence accuracy analysis.), how is the correct correspondence ratio (Correct/THW) calculated.
  2. Appendix D (Cross-backbone analysis, Fig. 14) answers the question of whether the anchor blocks exists on other backbone, but still, it is not enough not support the general conclusion that the proposed techniques can improve the performance of the different baseline models.
  3. As shown in Sec. D (Geometric correspondence accuracy analysis, Fig. 13), it indicates the trade-off between the "stronger correspondance" and "enough following blocks to inject guidance". It would be more informative if the authors can provide a curve to quantatively describe this trade-off.

1. Calculation of Correct/THW.

The video latent contains THWTHW tokens, while the reference latent contains HWHW tokens. After concatenation, the DiT self-attention matrix is W∈R(HW+THW)×(HW+THW)\mathbf{W}\in\mathbb{R}^{(HW+THW)\times(HW+THW)}. We extract the video-to-all attention weights Wv∈RTHW×(HW+THW)\mathbf{W}_v\in\mathbb{R}^{THW\times(HW+THW)}. For each video token ii, we identify its most attended token Pi=arg⁡max⁡jWv[i,j]P_i=\arg\max_j\mathbf{W}_v[i,j]. We retain video-to-reference matches where Pi<HWP_i<HW. Each matched token pair (i,Pi)(i,P_i) corresponds to two image patches. Using camera poses and depth-derived pixel correspondences, we consider a match correct if at least one valid pixel correspondence exists between the two patches. Let Correct\mathrm{Correct} denote the number of video tokens that are matched correctly. We compute Correct/THW\mathrm{Correct/THW}. The denominator is the total number of video tokens. Thus, Correct/THW measures the coverage of geometrically correct attention-derived correspondences.

With NRM, we first select an anchor frame and establish its correspondences with the reference image. Depth warping then propagates these correspondences to the other video frames, giving each supported video token a corresponding reference location. We count video tokens with geometrically correct matches and divide by THW, as before.

2. Cross-backbone analysis.

We added qualitative and quantitative comparisons on VACE-LTX (Appendix D, Fig. 14 and Table 6), evaluating ReCoG against FT and UCPE.

MethodTransErr↓RotErr↓FID↓FVD↓LPIPS↓SSIM↑DS-Max↑DS-Avg↑
+FT0.30055.7864182.171376.270.49660.45860.73930.6990
+UCPE0.18474.4088207.701363.070.52600.44190.68580.6440
Ours0.10462.1278142.78845.100.43770.48950.79590.7401

ReCoG achieves the best performance across all reported metrics, including camera accuracy, video quality, and reference consistency. Together with our results on VACE-Wan, these findings provide further evidence that the effectiveness of our correspondence-driven geometric guidance extends beyond the original backbone.

3. Quantitative trade-off in Anchor Block Selection.

We appreciate the reviewer's suggestion. Fig. 13 and Table 2 already provide complementary quantitative evidence for the trade-off between correspondence quality and subsequent guidance capacity, supporting our Anchor Block selection.

More importantly, our core contribution is to reify endogenous correspondences into explicit geometric guidance for camera-controllable R2V generation, whose effectiveness has been validated by extensive experiments. Identifying a potentially better Anchor Block may further improve performance, but would not change this central conclusion.

A denser block-wise sweep would require full fine-tuning and evaluation for each additional Anchor Block configuration, which is computationally expensive and difficult to complete within the current time and resource constraints. We believe the existing evidence sufficiently supports our design rationale and central contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.