- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CoFrame, a framework for improving real-world video retaking through synchronized reframing supervision. The authors identify a limitation of existing conditional video re-rendering methods: their reliance on synthetic multi-camera training data may restrict their ability to preserve diverse real-world appearances and subject dynamics.
To address this issue, CoFrame constructs synchronized source-target video pairs from real monocular recordings using spatial cropping and resizing. These transformations are represented as changes in camera intrinsics, allowing the resulting videos to be incorporated into camera-conditioned training alongside synthetic multi-camera data. While synthetic pairs provide supervision for extrinsic viewpoint changes, real-video reframings introduce complementary real-world appearance and motion supervision. The framework also introduces paired flow matching to supervise differences between camera-conditioned target predictions and a relative camera adapter to explicitly incorporate source-target camera relationships.
Experiments on DAVIS and DyCheck iPhone demonstrate improvements in camera trajectory accuracy, local source-target correspondences, and the proposed Motion Magnitude Error (MME). The authors further investigate the individual components through cumulative ablations and evaluate performance under limited synthetic multi-camera supervision.
Critical strengths
S1. A practical strategy for incorporating real-world motion and appearance into conditional video re-rendering.
The paper addresses an important limitation of video retaking methods trained predominantly on synthetic multi-camera data: the limited diversity of real-world appearances and subject dynamics. The proposed synchronized reframing strategy provides a practical way to construct temporally aligned training pairs from monocular videos without requiring synchronized real multi-camera recordings.
Representing crop-and-resize transformations through camera intrinsics allows the constructed data to be incorporated into existing camera-conditioned models while preserving the recorded temporal dynamics. Although intrinsic reframing cannot replace supervision for genuine extrinsic viewpoint changes, combining it with synthetic multi-camera data is a reasonable and practically useful direction.
S2. Encouraging improvements in motion preservation and camera accuracy.
The experiments demonstrate consistent improvements over several conditional re-rendering baselines on the main DAVIS evaluation. In particular, CoFrame reduces translation and rotation errors relative to ReDirector while improving local source-target correspondences and the proposed motion magnitude metric. The component ablations are also informative. Adding real-video reframings substantially improves motion-related metrics, while paired flow matching and relative camera conditioning further improve camera accuracy.
The evaluation on DyCheck iPhone provides additional evidence through comparison with recorded target views, and the limited-synthetic-data experiments suggest that real-video reframing can complement expensive multi-camera supervision.
Critical weaknesses
The proposed Motion Magnitude Error requires stronger validation, and the evidence for real-world motion preservation remains limited.
A major motivation of the paper is improving subject motion preservation under camera viewpoint changes. To evaluate this, the authors introduce Motion Magnitude Error (MME), which compares average residual image-plane motion magnitudes between source and generated videos.
However, the metric relies on subtracting a global median displacement to approximate camera-induced motion. Under substantial camera translation, rotation, or depth variation, camera-induced optical flow is spatially varying and cannot generally be removed by a single global displacement estimate.
Furthermore, source and generated tracks are matched primarily according to their residual motion directions, with matches selected independently for each source track. This does not guarantee physical point correspondence and may produce ambiguous many-to-one matches. The metric also compares average motion magnitudes rather than complete motion trajectories, making it potentially insensitive to temporal misalignment, motion-direction changes, and other forms of action inconsistency.
The evaluation on DyCheck iPhone offers a more direct comparison with recorded target views, but it includes only seven scenes, and the PSNR improvement over ReDirector is relatively modest (11.41 versus 11.12 dB). Without further validation of MME and more detailed per-scene results, it remains difficult to determine how reliably the reported improvements translate into better preservation of real subject motion under challenging novel-view transformations.
Questions for authors
Q1. How reliable is MME under substantial viewpoint changes and complex subject motion?
The proposed MME uses global median displacement to approximate camera-induced motion and matches source and generated trajectories based on residual motion-direction similarity. How sensitive is this metric to camera translation, scene depth variation, inaccurate tracking, and multiple moving subjects?
Could the authors validate MME on synchronized multi-view videos with known dynamic trajectories, or provide controlled perturbation experiments involving motion scaling, direction changes, and temporal misalignment? Such an evaluation would strengthen the interpretation of MME as a measure of subject motion preservation.
Q2. Can the authors provide more detailed evidence for real-world novel-view generalization?
The DyCheck iPhone evaluation provides direct ground-truth target-view comparisons, but the reported results are averaged over only seven scenes. Could the authors report per-scene PSNR and LPIPS, together with uncertainty estimates or paired statistical comparisons? It would also be helpful to analyze performance according to the magnitude of the source-target camera displacement and the amount of disocclusion. These results would help determine whether the improvements are consistent across scenes or concentrated in particular camera-motion regimes.
Thank you for the detailed responses. The authors have adequately addressed my concerns: the difference from InfCam is now clear (cropping is used to obtain supervision from real videos, not to augment synthetic data), the paired flow-matching loss is better motivated and supported by Table 3, and the empty-prompt experiment and crop bounds resolve the small-crop concern.
Overall, I appreciate the authors' efforts, and I will maintain my score (3: Weak acceptance).