acceptodds
Withdrawn from ICLR 2027

Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coarse-to-real (C2R) video generation converts coarse animation, simulation, or game-rendered videos into photorealistic videos while preserving scene layout, camera motion, and dynamics. Training such models requires aligned (coarse, realistic) video pairs, which are expensive to create and confine prior systems to narrow synthetic domains. We exploit a simple asymmetry: paired videos are hard to obtain, but paired real-to-coarse images are easy to construct with existing editing and 3D reconstruction tools. Using these image pairs, we train the DINO-R2C model to transfer realistic images to coarse images in DINO feature space, and apply it frame-wise to ordinary real videos to synthesize pseudo-coarse videos. The C2R# model then learns to generate each real video from its pseudo-coarse counterpart, free from needing (coarse, realistic) video pairs. At inference, the DINO-R2C model is discarded and actual coarse videos drive the C2R# model directly. Thus, our pair-free training scales with any real-video corpora rather than with paired synthetic videos.

Primary area:
generative models
Keywords:
AI rendering, Controllable video generation, Synthetic data generation, Cross-domain video translation, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nhy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes C2R#, which uses paired image supervision to address the difficulty of obtaining paired videos for coarse-to-real video generation. The authors train DINO-R2C to convert real videos frame by frame into pseudo-coarse videos, then mitigate information leakage through RGB decoding and re-encoding.

Critical strengths
  1. The idea of constructing pseudo-paired videos based on DINO features is reasonable, and the RGB bottleneck is also designed to address the information differences between training and inference.
  2. The method is evaluated on three types of inputs: games, geometric scenes, and animations, and performs well on the DOVER and VEFX metrics.
Critical weaknesses
  1. The ablations of key design choices are insufficient. Only qualitative examples are provided for the RGB bottleneck, with no quantitative results; the camera augmentation ablation also does not demonstrate a positive effect on final video generation performance. The relevant text may need to be reorganized.
  2. In Table 2, both optical flow and camera trajectory alignment are worse than those of VACE 2.2, and the VLM realism score is also slightly lower than that of the original C2R. This is very strange and undermines the paper's conclusions.
  3. The games and geometric scenes mainly come from Roblox, while the animations are generated by Wan2.2. Only aggregate metrics are reported, with no domain-specific results, which is insufficient for validation.
Questions for authors
  1. Could the authors provide quantitative ablations of the RGB bottleneck and camera augmentation on the final performance of C2R#?
  2. How large is the domain gap between pseudo-coarse videos and actual coarse inputs?
  3. Can the method maintain its performance on videos from other game engines or manually created animations?
Reviewer_Y8csOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the task of coarse-to-real video generation. Since high-quality paired coarse–real videos are difficult to collect in practice, the authors explore using paired image data as an alternative source of supervision. Specifically, they train a DINO-based real-to-coarse mapping with an RAE decoder to transform real images into coarse counterparts. This module is then applied frame by frame to real videos to produce pseudo-coarse videos, which are paired with the original videos to train the C2R generator. In this way, the method removes the need for paired coarse–real videos and enables pair-free training at the video level. The method is evaluated on game renders, geometry blockouts, and stylized animations.

Critical strengths
  1. The pair-free idea is valuable. In the R2C pipeline, the authors train an image-level mapping that projects real video frames into the coarse video space, enabling C2R training without paired coarse–real videos.
  2. Decoding and then re-encoding the coarse DINO features is a good design choice. The decoder acts as a filter that removes information in the predicted features that is unavailable in actual coarse inputs and may not be useful for downstream control.
Critical weaknesses
  1. The R2C model is still an image-level mapping. When applied frame by frame to videos, it produces visible flickering, which can also be observed in the supplementary results. Although the authors argue that the target videos are high-quality and temporally coherent, the effect of noisy control inputs cannot be ignored, since the input condition plays a critical role in controllable generation. It also introduces a train–test gap because actual coarse videos do not contain this type of frame-wise flickering, potentially limiting the performance ceiling.
  2. The DINO-based condition injection may limit control accuracy. The method encodes the input video into DINO features and injects them into the DiT. A more common strategy in video generation is to encode the input into VAE latents and fuse them through in-context concatenation, which may preserve more information from the source video and improve alignment. Although the weaker control alignment results in Table 2 may partly reflect a video quality and control trade-off, the DINO-based conditioning pathway could also be a reason for the lower control precision.
Questions for authors
  1. Directly injecting the predicted DINO-R2C features causes the generated video to collapse. Does this also indicate that the predicted features are inaccurate or substantially different from DINO features extracted from actual coarse videos? If the feature-space discrepancy were small, one might expect moderate degradation rather than complete failure.
  2. Could the authors discuss the difference between conditioning on DINO features and conditioning on VAE latents through in-context concatenation, particularly regarding information preservation and control alignment? If possible, an ablation comparing the two conditioning strategies would be helpful, though it is not essential.
Reviewer_c7qoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a coarse-to-real video generation framework that replaces paired coarse–real video supervision with paired image supervision. An image-level model, DINO-R2C, converts real video frames into pseudo-coarse controls, which are then paired with the original videos to train control adapters for a frozen Wan2.2 backbone. A decoding and re-encoding bottleneck reduces information leakage from the real inputs. The method is trained on approximately 390K real videos and evaluated on 300 coarse videos spanning game renders, geometry blockouts, and stylized animations. Experiments demonstrate competitive visual quality, alongside ablations of pseudo-coarse generation and experiments transferring the generated training pairs to other models.

Critical strengths

A practical approach to reducing the paired-video data bottleneck. Using image-level supervision to construct video training pairs is a compelling idea, since aligned coarse–real videos are expensive to obtain. Separating supervision acquisition from the real-video corpus provides a potentially scalable way to support multiple coarse input styles.

Strong output-quality results with a relatively lightweight adaptation. The proposed model trains only control and fusion adapters and achieves the best reported DOVER and VEFX scores among the evaluated methods. Comparisons with baselines adapted on the same generated data are also valuable for examining whether the benefits extend beyond the proposed architecture.

Critical weaknesses

The temporal reliability of the pseudo-coarse training data is not sufficiently established. DINO-R2C processes frames independently, without explicit temporal conditioning or a temporal consistency objective. Camera augmentation improves the reported stability metrics, but does not directly constrain consistency through articulation, occlusion, or changes in object visibility. Moreover, the temporal ablations cover only the coarse-geometry modality. The paper argues that training against coherent real-video targets teaches C2R# to tolerate residual jitter, but does not directly establish how this jitter affects final video quality or control fidelity. This matters because inconsistent controls could encourage the model to discount input changes, potentially contributing to the reported trade-off between smooth outputs and faithful motion preservation.

Evidence that the generated pairs provide broadly useful supervision is mixed. Fine-tuning Editto Sim2Real improves all ten reported metrics, but adapting VACE 2.2 improves only DINO correspondence and edit exclusivity while degrading the other eight metrics, including realism from 61.88 to 51.96. These results demonstrate utility for one recipient model, but leave substantial uncertainty about the data’s general applicability. The current experiments do not clearly distinguish limitations of the pseudo-coarse data from limitations of the adaptation configuration. Since reusable, scalable training supervision is central to the paper’s contribution, understanding this discrepancy is important to assessing its significance.

Questions for authors

Clarification on temporal consistency: Beyond the coarse-geometry evaluation, have you measured temporal stability across the other three pseudo-coarse modalities, particularly for articulated motion and occlusions? Do the observed inconsistencies primarily involve appearance flicker, or do they also alter geometry and motion?

Evidence supporting tolerance to jitter: Do you have downstream results comparing C2R# trained with pseudo-coarse data from different DINO-R2C augmentation settings? Such a comparison would help establish whether improved pseudo-coarse consistency translates into better output quality and control alignment, rather than only better intermediate metrics.

Clarification on transfer experiments: What parameters, optimization budgets, and model-selection procedures were used to adapt Editto and VACE? Were these configurations validated separately for each model, and what evidence explains VACE’s degradation on eight of ten metrics? This distinction would materially affect whether I interpret the results as a limitation of the generated supervision or of the fine-tuning setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.