acceptodds
Withdrawn from ICLR 2027

FilmTogether: Cinematic Video Generation from Low-Cost Inputs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The widespread adoption of mobile devices has democratized video recording, but a stark aesthetic divide persists between casual, low-cost captures and high-cost cinematic productions. Bridging this gap requires upgrading heterogeneous production elements, such as lighting, specialized costumes, and materials, without altering the core layout, identity, and temporal motion of the original footage. However, this task is heavily bottlenecked by two important challenges: 1) Lacking a high-quality paired training data specifically designed to handle complex low-to-high appearance shifts. 2) Ignoring the capacity of current video editing models to perceive and enhance the multiple low-cost compositions. To address these issues, we present CineCraft-120K, a large-scale dataset of aligned low- and high-cost video pairs annotated with pair-aware editing prompts. CineCraft-120K features a diverse, high-resolution and high-cost Cinematic Crafting bank spanning decades of media production, meticulously taxonomized into 5 broad domains and 23 fine-grained sub-categories. It also provides element-level mapping between low- and high-cost counterparts, offering pair-aware rich textual prompts that precisely detail the required aesthetic transformations while anchoring structural layout. Architecturally, we propose a simple yet effective dual-memory feature adapter, where a text-memory expands semantic capacity for low2high appearance changes, and a hint memory supplies layer-wise visual feature corrections of high-cost element before context injection. Extensive experiments and ablation studies demonstrate the superiority of CineCraft-120K for low-to-high-cost video editing, alongside the effectiveness of our dual-memory adapter.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CINEMATIC VIDEO GENERATION

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_GQZ8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose a FilmTogether, a video generative system for transforming low cost captured videos in to high-cost cinematic productions. At the core of the method is two main contributions: 1. CineCraft-120k, a large scale dataset of aligned low and high cost video pairs annotated with editing prompts, synthetically generated by transforming high quality production ready datasets into low quality videos. 2. instruction-routed text memory and layer-wise hint memory which enables such low quality to high quality video. Temporal consistency, PickScore, and VLM judge scores are provided on the synthetic datasets. User study is also performed.

Critical strengths
  1. The dataset is very large, and building it takes about 1500 gpu-days, which cannot be easily reproduced by many groups. If released, this can be useful to the community.
Critical weaknesses
  1. I am concerned about a lack of qualitative examples of the technique and the dataset. Across the entire paper, only 4 qualitative examples are shown which are output of the technique. Unfortunately this is not enough for me to build sufficient intuition about what the performance of the proposed method or the quality of the dataset is like. Therefore, I am strongly uncertain about the quality of the work and cannot recommend acceptance.
  2. Evaluations are all in domain on the proposed synthetic dataset. It is not clear if the technique will generalize to open domain videos, which is very critical for modern video generative methods.
  3. Even on the in-domain dataset, the quantitative performance often very low or questionable. The temporal consistency metrics (CLIP-F, DINO-F) is all very saturated for all tested methods. On the other hand, pickscore is only 18.83 and lags behind Bernini-R-14B(19.05). Moreover, the pickscores of all methods are very low. Why?
  4. Only CLIP-F, DINO-F ablation scores are provided for the proposed method, and the improvement is extremely marginal. This only suggests slightly improved temporal consistency, but not editing quality. It does not seem to support author's claim that "text-memory expands semantic capacity for low2high appearance changes, and a hint memory supplies layer-wise visual feature corrections of high-cost element before context injection".
  5. it is not clear what is the motivation for converting low cost movie into high cost movie vs e.g. reference subject motion driven video generation (e.g. WAN-Animate-2) or identity conditioned video generation (such as WildActor Unconstrained Identity-Preserving Video Generation). In fact I think the author should benchmark against these two categories of techniques.
Questions for authors

The biggest difference between low cost captures vs professional productions might be the storyboard, the footage cuts, and the camera poses, which the proposed method cannot handle. I wonder what is the authors thought on that? Why is the proposed method still very useful despite such shortcomings?

Reviewer_evGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FilmTogether, a framework for transforming low-cost videos into cinematic-quality videos while preserving their original content and motion. The main contribution is to introduce CineCraft-120K, a dataset containing over 120K paired low- and high-cost videos with detailed editing instructions. The dataset is constructed by collecting high-quality cinematic footage and synthesizing corresponding low-cost versions through first-frame editing and depth-guided video generation. To improve cinematic editing, the authors propose a dual-memory adapter consisting of an instruction-routed text memory and a layer-wise hint memory, which enhance semantic conditioning and visual feature adaptation, respectively. Experiments demonstrate improvements over existing video editing methods in aesthetic quality, temporal consistency, and human preference.

Critical strengths
  1. The paper addresses an interesting and practical video editing task that requires coordinated modifications to lighting, costumes, materials, and scene appearance while preserving the original video structure. The proposed dataset construction pipeline (starting from high quality videos and synthesizing degraded videos) is a reasonable idea.

  2. The dual-memory adapter provides a lightweight way to enhance both textual and visual conditioning without modifying the pretrained diffusion backbone.

Critical weaknesses
  1. The low-cost videos in CineCraft-120K are synthetically generated from high-quality cinematic videos, rather than collected from real low-budget productions or everyday recordings. I'm not very sure if this pipeline meets the central claim in the paper that the method "bridges the gap between casual, low-cost captures and highcost cinematic productions." I think the author should at least consider some imaging aspects of the low-quality videos: e.g., synthetic degradation to simulate high ISO noise, overexposure, motion blur or compression artifacts that can actually happen in the recordings of everyday devices.

  2. The effectiveness of the proposed dual-memory adapter is not sufficiently validated. The component ablation in Table A2 only reports CLIP-F and DINO-F, with almost identical results across different settings, and does not include the aesthetic metrics that are central to the paper's claims. Moreover, the full-model results in Table A2 (0.9934/0.9939) are inconsistent with the results in Table 1 (0.9967/0.9985). Additional ablations on aesthetic quality and instruction following are necessary to demonstrate the individual contributions of the two memory modules.

  3. The evaluation relies heavily on the proposed VLM-Aesthetic metric, which is obtained by distilling aesthetic judgments from Gemini 3.1 Pro into Qwen3-VL. However, the paper does not provide evaluations or analysis of whether this evaluator actually reflects human preferences. Furthermore, CLIP-F and DINO-F measure adjacent-frame similarity rather than whether the edited videos accurately preserve the source motion and content.

  4. The human evaluation is quite limited with only 10 participants and 10 video examples. The paper also lacks failure-case analysis, particularly for challenging inputs involving complex motion, occlusions, and substantial appearance transformations.

Questions for authors

See weaknesses

Reviewer_jTNQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FilmTogether for the task of transforming low-cost videos into high-quality cinematic videos. The authors introduce CineCraft-120K, a dataset consisting of 125K high-cost and low-cost video pairs. Specifically, they use Qwen2.5-VL to identify high-cost visual elements and generate high-to-low editing instructions. They then generate a low-cost first frame based on their instruction and use it to synthesize the corresponding low-cost video. Through this pipeline, the authors construct paired high-cost and low-cost videos. Additionally, this paper introduces instruction-routed text memory and layer-wise hint memory as two architectural modifications, to adapt the model to this transformation task. Experimental results show that FilmTogether outperforms existing video editing methods.

Critical strengths
  • The paper focuses on the task of transforming low-cost videos into high-quality cinematic videos, which has significant practical value.

  • The paper introduces CineCraft-120K, a large-scale dataset designed for this task. The dataset contains approximately 125K paired low-cost and high-cost videos across 23 fine-grained subcategories. Experimental results demonstrate that CineCraft-120K is more effective than existing datasets, such as Ditto-1M, for this task.

Critical weaknesses
  • My main concern lies in Section 3.

    1. The proposed methods are relatively simple and lack clear and convincing intuition.
      1. For Text Memory, the authors motivate their design by stating that "The frozen text encoder provides general-purpose instruction features but cannot directly adapt its parameters to cinematic enhancement." Their solution is to introduce four learnable tokens, use a two-layer MLP to compute gating values, and append the four gated features to the original text features. However, a more straightforward approach would be to directly fine-tune the text encoder or apply LoRA to it. The authors do not sufficiently justify why their proposed design is preferable to these alternatives.
      2. For Layer-Wise Hint Memory, the authors aim to separate structural information from low-cost appearance features. However, they do not clearly explain how the proposed design addresses this motivation. Moreover, the Layer-Wise Hint Memory corrections depend only on textual features, rather than the visual hints themselves, which makes the connection between the proposed method and its motivation less convincing.
    2. The paper lacks sufficient ablation studies to demonstrate the effectiveness of the proposed modifications or comparisons with more straightforward alternatives (e.g., directly fine-tuning the text encoder). The only ablation results are presented in Table A2, where the evaluation is limited to CLIP-F and DINO-F. These metrics are insufficient to evaluate the effectiveness of the proposed modules. Besides, the full model even does not show a meaningful improvement over the variants without the two memory modules on these two metrics.
  • The evaluation protocol has several limitations.

    1. The current evaluation metrics do not explicitly measure how well the generated videos preserve the content and structure of the original low-cost videos.
    2. The proposed Distilled Aesthetic Evaluator itself is not adequately validated. The paper does not provide sufficient evidence of its reliability or its agreement with human judgments.
    3. The authors evaluate their method on CineCraft-Bench, where the low-cost input videos are synthetically generated. It will be better to evaluate the proposed method on eal-world low-cost videos.
  • The quality of the automatically constructed dataset is also not sufficiently validated. Since CineCraft-120K is generated through an automated pipeline, I wonder whether the authors have conducted any systematic quality checks of the resulting video pairs and editing instructions. Such validation would be important to establish the reliability of the dataset.

Questions for authors

Please see the weaknesses.

Reviewer_763mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FilmTogether, a framework for transforming low-cost videos into cinematic-quality outputs while preserving source content and motion. The authors construct CineCraft-120K, a large-scale paired video dataset, and propose a dual-memory adapter to improve instruction-based cinematic editing.

Critical strengths
  1. Large-scale dataset contribution: The paper introduces CineCraft-120K, comprising approximately 120K paired low- and high-cost videos with detailed editing instructions. The dataset provides a potentially valuable resource for cinematic video editing research.

  2. Comprehensive experimental comparisons: The authors compare their method against 12 existing video editing approaches and report improvements in aesthetic quality and temporal consistency. The human preference study further supports the effectiveness of the proposed framework.

Critical weaknesses
  1. The ablation results in Table A2 show negligible differences between the full model and variants without the proposed memory components. Furthermore, component-wise ablations on the primary aesthetic metrics are missing, making it difficult to establish the contribution of each component.

  2. The VLM-Aesthetic evaluator is distilled from another VLM, but its agreement with human judgments is not sufficiently validated. This raises concerns about the reliability of the reported aesthetic improvements.

  3. CineCraft-120K constructs low-cost videos synthetically from cinematic footage, which may not reflect the characteristics of authentic low-budget recordings. The paper provides limited quantitative evidence of generalization to independently collected real-world videos.

  4. CLIP-F and DINO-F measure adjacent-frame similarity rather than directly evaluating whether the original motion and structure are preserved. More direct evaluations of motion fidelity, identity consistency, and spatial preservation are needed.

  5. The user study includes only ten participants and ten video cases, limiting the diversity of the evaluation. Confidence intervals and inter-rater agreement are also not reported.

Questions for authors
  1. How are the four memory codes in the Text Memory and Hint Memory selected? Is there a specific reason for choosing four codes, and how sensitive is the performance to the memory size?

  2. The Hint Memory applies a shared instruction-conditioned residual across all spatial and temporal tokens at each layer. How does this design support localized appearance changes, such as modifying a specific object's material or costume, without affecting unrelated regions?

  3. The dataset covers cinematic footage released between 1970 and 2023. How does the model handle potentially conflicting cinematic styles across different eras, and can users explicitly control the desired cinematic style?

  4. The training procedure uses a reference-frame prior that is gradually annealed and disabled during inference. What motivates this training strategy, and how does it affect the model's ability to preserve source content without reference-frame guidance?

Reviewer_1XGeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

FilmTogether studies cinematic enhancement of ordinary videos while preserving identity, layout, and motion. It introduces CineCraft-120K by synthesizing low-cost counterparts of professional footage and annotating the reverse transformations. Built on VACE with Ditto’s training recipe, the model adds instruction-gated text tokens and layer-wise hint residuals. Evaluation combines temporal feature similarity, a distilled aesthetic judge, and human preferences.

Critical strengths
  • Using real cinematic footage as enhancement targets provides useful supervision for lighting, costumes, and materials without relying entirely on generated targets.
  • Table 2 supports the dataset’s value: CineCraft-120K improves VLM-Aesthetic from 3.22 to 3.78 for Ditto and from 3.66 to 3.87 for FilmTogether.
  • The anonymized, synchronized user study provides additional evidence of perceptual improvements, with FilmTogether preferred on the ten evaluated cases.
Critical weaknesses
  • The adapter’s benefit is not convincingly established. On CineCraft-120K, FilmTogether improves aesthetic score over Ditto by only 0.09, with no improvement in CLIP-F, DINO-F, or PickScore. Table A2 reports almost unchanged temporal scores after removing either memory, but omits aesthetic and instruction-following results, the capabilities these modules are intended to improve.
  • The mechanism does not directly support the claimed element-level correction. In Eqs. (3–4), Hint Memory depends only on pooled text and broadcasts one residual across all source tokens at each layer. It functions as a text-conditioned layer bias. Comparisons with parameter-matched conditional biases and ungated text tokens are needed to establish the value of memory routing.
  • The automatic metrics provide limited evidence of preservation. CLIP-F and DINO-F measure adjacent-output similarity and can reward reduced motion. The aesthetic judge sees only the first two sampled output frames, without the source video, and its agreement with held-out human judgments is unreported. The ten-case user study only partially addresses these limitations.
  • Pair alignment and transfer to real low-cost footage remain insufficiently validated. Depth-guided synthesis does not guarantee preservation of facial identity or fine motion, and revising captions cannot repair such mismatches. Quantitative pair-quality checks and a clearly identified real-capture benchmark are needed; the two everyday-video examples are encouraging but limited.
Questions for authors
  • Which checkpoints and evaluation subsets produce Tables 1, 2, and A2? Why does the aesthetic protocol mention 50 rather than 100 videos?
  • Do both memories improve aesthetics and instruction following over parameter-matched conditioning baselines under identical training?
  • How were the evaluator’s 5,000 training clips selected and separated from the benchmark? What is its agreement with independent human ratings?
  • Are splits disjoint by source film or series, and how are identity/motion mismatches in synthesized pairs filtered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.