acceptodds
Under review as a conference paper at ICLR 2027

CoFrame: Learning Real-World Video Retaking via Synchronized Reframing

Abstract

Video retaking aims to synthesize a new video that follows an input camera trajectory while preserving the appearance and motion depicted in the input video. Enabled by advances in conditional video generation models, together with synthetic multi-camera supervision, recent works have shown strong retaking performance, especially in terms of camera accuracy. However, faithfully preserving subject appearance and dynamics in real-world videos remains challenging, particularly for rapid movements and large displacements. A key challenge is still the lack of real synchronized multi-view recordings for training. In this work, we address this bottleneck from the views of complementary supervision. Specifically, we introduce CoFrame, a framework that constructs synchronized reframings from real monocular videos to complement synthetic multi-camera supervision for conditional re-rendering. Our key insight is that synthetic multi-camera pairs supervise extrinsic viewpoint changes, while synchronized real-video reframings provide complementary supervision for preserving real appearance and dynamics, with camera intrinsics as the bridge for expressing crop-and-resize transformations as explicit camera conditions. We further explore relative camera conditioning and paired flow matching to encode source-target camera relationships and jointly supervise synchronized target predictions. Experiments demonstrate improved motion preservation and camera accuracy over ReCamMaster, ReDirector, and Geo-Align, together with improved data efficiency under limited synthetic multi-camera supervision.

Primary area:
generative models
Keywords:
Generative model, Video Retaking, Video Diffusion, Video Editing, Motion Preservation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cvgFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoFrame, a framework for improving real-world video retaking through synchronized reframing supervision. The authors identify a limitation of existing conditional video re-rendering methods: their reliance on synthetic multi-camera training data may restrict their ability to preserve diverse real-world appearances and subject dynamics.

To address this issue, CoFrame constructs synchronized source-target video pairs from real monocular recordings using spatial cropping and resizing. These transformations are represented as changes in camera intrinsics, allowing the resulting videos to be incorporated into camera-conditioned training alongside synthetic multi-camera data. While synthetic pairs provide supervision for extrinsic viewpoint changes, real-video reframings introduce complementary real-world appearance and motion supervision. The framework also introduces paired flow matching to supervise differences between camera-conditioned target predictions and a relative camera adapter to explicitly incorporate source-target camera relationships.

Experiments on DAVIS and DyCheck iPhone demonstrate improvements in camera trajectory accuracy, local source-target correspondences, and the proposed Motion Magnitude Error (MME). The authors further investigate the individual components through cumulative ablations and evaluate performance under limited synthetic multi-camera supervision.

Critical strengths

S1. A practical strategy for incorporating real-world motion and appearance into conditional video re-rendering.

The paper addresses an important limitation of video retaking methods trained predominantly on synthetic multi-camera data: the limited diversity of real-world appearances and subject dynamics. The proposed synchronized reframing strategy provides a practical way to construct temporally aligned training pairs from monocular videos without requiring synchronized real multi-camera recordings.

Representing crop-and-resize transformations through camera intrinsics allows the constructed data to be incorporated into existing camera-conditioned models while preserving the recorded temporal dynamics. Although intrinsic reframing cannot replace supervision for genuine extrinsic viewpoint changes, combining it with synthetic multi-camera data is a reasonable and practically useful direction.

S2. Encouraging improvements in motion preservation and camera accuracy.

The experiments demonstrate consistent improvements over several conditional re-rendering baselines on the main DAVIS evaluation. In particular, CoFrame reduces translation and rotation errors relative to ReDirector while improving local source-target correspondences and the proposed motion magnitude metric. The component ablations are also informative. Adding real-video reframings substantially improves motion-related metrics, while paired flow matching and relative camera conditioning further improve camera accuracy.

The evaluation on DyCheck iPhone provides additional evidence through comparison with recorded target views, and the limited-synthetic-data experiments suggest that real-video reframing can complement expensive multi-camera supervision.

Critical weaknesses

The proposed Motion Magnitude Error requires stronger validation, and the evidence for real-world motion preservation remains limited.

A major motivation of the paper is improving subject motion preservation under camera viewpoint changes. To evaluate this, the authors introduce Motion Magnitude Error (MME), which compares average residual image-plane motion magnitudes between source and generated videos.

However, the metric relies on subtracting a global median displacement to approximate camera-induced motion. Under substantial camera translation, rotation, or depth variation, camera-induced optical flow is spatially varying and cannot generally be removed by a single global displacement estimate.

Furthermore, source and generated tracks are matched primarily according to their residual motion directions, with matches selected independently for each source track. This does not guarantee physical point correspondence and may produce ambiguous many-to-one matches. The metric also compares average motion magnitudes rather than complete motion trajectories, making it potentially insensitive to temporal misalignment, motion-direction changes, and other forms of action inconsistency.

The evaluation on DyCheck iPhone offers a more direct comparison with recorded target views, but it includes only seven scenes, and the PSNR improvement over ReDirector is relatively modest (11.41 versus 11.12 dB). Without further validation of MME and more detailed per-scene results, it remains difficult to determine how reliably the reported improvements translate into better preservation of real subject motion under challenging novel-view transformations.

Questions for authors

Q1. How reliable is MME under substantial viewpoint changes and complex subject motion?

The proposed MME uses global median displacement to approximate camera-induced motion and matches source and generated trajectories based on residual motion-direction similarity. How sensitive is this metric to camera translation, scene depth variation, inaccurate tracking, and multiple moving subjects?

Could the authors validate MME on synchronized multi-view videos with known dynamic trajectories, or provide controlled perturbation experiments involving motion scaling, direction changes, and temporal misalignment? Such an evaluation would strengthen the interpretation of MME as a measure of subject motion preservation.

Q2. Can the authors provide more detailed evidence for real-world novel-view generalization?

The DyCheck iPhone evaluation provides direct ground-truth target-view comparisons, but the reported results are averaged over only seven scenes. Could the authors report per-scene PSNR and LPIPS, together with uncertainty estimates or paired statistical comparisons? It would also be helpful to analyze performance according to the magnitude of the source-target camera displacement and the amount of disocclusion. These results would help determine whether the improvements are consistent across scenes or concentrated in particular camera-motion regimes.

Reviewer_bc6DOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CoFrame, a video-retaking framework that combines synchronized reframings of real monocular videos with synthetic multi-camera supervision. Crop-induced intrinsic changes provide camera-conditioned training pairs that preserve real-world appearance and motion. The framework additionally incorporates paired flow matching and relative camera conditioning to improve camera control.

Critical strengths
  1. The paper is well motivated, addressing the lack of synchornized real-world novel-view video dataset.
  2. Paired flow matching loss looks interesting, but a more intuitive explanation (or related paper) of how it helps optimize the model would be helpful.
  3. Improving model performance through augmentation looks promising.
  4. The proposed model clearly outperforms other baselines in both quantitative and qualitative comparisons (and the supplementary videos as well).
Critical weaknesses
  1. A similar approach to camera intrinsic augmentation was proposed in [1]. It includes spatial augmentation, such as cropping, and temporal augmentation through video reversal to allow different first frames. The differences between this paper and [1] should be discussed.

  2. Clearer motivation and explanation are needed for the paired flow-matching loss. Since the standard flow-matching objective already encourages satisfying the paired objective, further evidence of its additional benefit would be helpful, for example, a comparison against simply adjusting the learning rate or loss weight.

  3. Improving performance through augmentation looks promising, but edge cases may introduce limitations. For example, a small crop may remove important context, making the cropped video inconsistent with the text prompt describing the entire video.

References: [1] https://arxiv.org/abs/2512.17040

Questions for authors

Please refer to weaknesses.

2 comments

Thank you for the detailed responses. The authors have adequately addressed my concerns: the difference from InfCam is now clear (cropping is used to obtain supervision from real videos, not to augment synthetic data), the paired flow-matching loss is better motivated and supported by Table 3, and the empty-prompt experiment and crop bounds resolve the small-crop concern.

Overall, I appreciate the authors' efforts, and I will maintain my score (3: Weak acceptance).

Thank you for your prompt response and thoughtful feedback. We are glad that our responses have adequately addressed your concerns, and appreciate your time and careful review.

Reviewer_5FXHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes CoFrame, which is a video generation method which focuses on video retaking. Previously many methods like ReCamMaster mainly rely on multi-view synthetic data. However, these data has issues like decreasing the motion. The key idea of this work is to construct synchronized reframings from real monocular videos. All of experiments are done on Wan2.1-1.3B and using benckmark of DAVIS and DyCheck iPhone. Ablations show that the reframing data mainly improves motion preservation, while the paired loss and the adapter mainly improve camera accuracy.

Critical strengths
  1. The visualization is wonderful. Thanks for your sum_mat and project page. Those videos help me to evaluate the quality of generated videos. Those videos are very clear to show your work is better than other works.

  2. This idea is intuitive and novel. Crop and resize is equivalent to a change in camera intrinsics is a basic fact. This work use this fact to address a real bottleneck and easily to generate much data.

  3. The cost of data is cheap and this idea is easy to scale up.

Critical weaknesses
  1. The largest issue is that the real-world data only provide “fake” viewpoint changes. Looking at the videos, the method rarely generates objects or scene content that are absent from the original video. It is therefore hard to tell whether the method can handle genuine 3D viewpoint changes.

  2. The main evaluation on DAVIS has no ground truth. All metrics rely on estimators such as ViPE, CoTracker, and GIM, so the errors and biases of these estimators are mixed into the results.

Questions for authors
  1. Could you please report results stratified by the magnitude of the requested camera motion (like small, medium, and large rotation/translation), to show whether the gains hold under large 3D viewpoint changes?

  2. I am not an expert in video generation, but I feel that many issues seen in models with small parameter counts do not appear in larger models. I am not sure whether this problem is also the case here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.