acceptodds
Withdrawn from ICLR 2027

CineRef: Cinematic Reference-Guided Audio-Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cinematic reference-guided audio-video generation aims to synthesize high-quality videos conditioned on reference images and audios under complex cinematic scenarios. However, current open-source methods primarily treat reference information as passive conditioning, making reference injection fragile under large camera motions and frequent shot transitions. To address this, we propose CineRef, a unified framework for cinematic reference-guided audio-video generation. The key insight is that reference information should be explicitly linked to the generated video during training and consistently preserved throughout the entire clip, rather than treated as passive and transient conditioning. Based on this insight, we first propose Dual-Anchor Reference Alignment, which combines a reference ID anchor for explicit ID supervision with a face-aware content anchor for preserving natural facial content. We then introduce Reference Trajectory Consistency to maintain long-range ID coherence across substantial viewpoint and scale variations along ID-wise trajectory. Finally, we design Audio-Visual Reference Binding to establish explicit correspondence between each visual ID and its corresponding audio reference. Extensive experiments on our proposed CineSet for audio-video generation demonstrate that CineRef achieves superior ID injection and preservation under diverse camera motions and shot transitions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reference-Guided Audio-Video Generation, Subject-Driven Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g7kyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CineRef, a reference-guided audio-video generation framework built on LTX-2.3 for cinematic scenes with large camera motions and shot transitions. It adds explicit ID supervision with a face-aware content loss, a trajectory-level ID consistency loss, and RoPE-based audio-visual reference binding. The authors also introduce CineSet, a 100-sample benchmark, and report improved face similarity on OpenS2V and CineSet.

Critical strengths
  1. On OpenS2V, FaceSim-Cur improves from 55.23% (second best) to 74.13%, and the gain over LTX-Ingredients with the same backbone is substantial.
  2. The data construction pipeline (cross-trace reference retrieval, TTS-augmented reference audio, structured captions) is thorough and practically useful.
  3. Open-source reference-guided joint audio-video generation is a meaningful direction.
Critical weaknesses
  1. The ArcFace ID loss and the mask-weighted flow-matching loss are common practices. Aligning audio markers via temporal RoPE is close to the synchronized RoPE in DreamID-Omni. L_traj is largely redundant with L_id, and since only four frames are sampled per clip with K_traj = 3, the trajectory set nearly coincides with the frames used for L_id.

  2. ArcFace is used for the ID loss and also for FaceSim-Arc evaluation, which risks metric gaming. FaceSim-Cur uses a different model but is highly correlated.

3.ID gains may come at the cost of motion and naturalness. On OpenS2V, MotionAmplitude is only 14.77%, NaturalScore (68.32%) is below RefAlign, VINO, and Kaleido, and GmeScore (65.04%) is among the lowest of open-source methods. The TotalScore lead comes almost entirely from FaceSim, which conflicts with the claim that the improvement is not due to copy-and-paste behavior.

Questions for authors
  1. Eq. 5 uses a generated frame as the trajectory anchor rather than the reference image. Why is this choice made, and how do you prevent the loss from propagating ID drift if the anchor itself deviates from the reference? Please also clarify the inconsistency with Fig. 2, where the trajectory anchor appears to be the reference image.

  2. ArcFace is used both in training and in FaceSim-Arc evaluation. Can you report ID similarity with an independent evaluator or a user study to rule out metric gaming?

  3. On OpenS2V, MotionAmplitude, NaturalScore, and GmeScore are relatively low. Does the stronger ID supervision reduce motion dynamics or naturalness?

  4. Can you provide supplementary videos (I don't know if it is allowed), especially for multi-shot and large camera motion cases?

Reviewer_sitNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CineRef, a framework for cinematic reference-guided audio–video generation that explicitly models and preserves reference identities across complex camera motion and shot transitions. The key components are Dual-Anchor Reference Alignment (an ID anchor with ArcFace plus a face-aware flow-matching content anchor to mitigate face-scale bias), Reference Trajectory Consistency (trajectory-level ID matching under large scale/viewpoint changes), and Audio-Visual Reference Binding (RoPE-aligned audio markers in the VAE branch and explicit visual–text binding via a VLM branch). Experiments on OpenS2V and a newly introduced CineSet benchmark indicate large gains in identity preservation and robust performance under multi-shot scenarios, with ablations supporting the contribution of each module.

Critical strengths
  1. The dual-anchor design is a clear and practical solution to a known failure mode of pure ID losses (face enlargement/copy-paste bias), combining identity fidelity with face-aware content constraints.
  2. Trajectory-level ID consistency directly targets long-range drift under large scale/motion and shot transitions, which are under-explored in open-source systems. Audio–visual binding via learnable audio markers temporally aligned with visual references is a simple but effective mechanism for slot-level multimodal correspondence; pairing it with VLM-based role binding is conceptually coherent.
  3. Audio–visual binding via learnable audio markers temporally aligned with visual references is a simple but effective mechanism for slot-level multimodal correspondence; pairing it with VLM-based role binding is conceptually coherent.
Critical weaknesses
  1. Heavy reliance on face detection/ArcFace may limit applicability beyond frontal human faces, across demographics, occlusions, or non-human subjects; failure handling is not discussed.
  2. Audio marker design and its interaction with temporal RoPE are described at a high level; more specifics (dimensions, insertion points, ablation on marker-only vs RoPE-only) would aid reproducibility.
Questions for authors
  1. CineRef substantially improves identity similarity, while its motion amplitude and naturalness scores remain below several baselines in Table 1. Do the identity and trajectory constraints sometimes restrict facial expressions or head movements when following dynamic prompts? A brief analysis of such cases would clarify how the method balances identity preservation with expressive, natural motion.
  2. What should I do if there are tracking errors in Trajectory Consistency?
  3. Will IDAnchor cause the face to be too similar to the reference image, such as becoming rigid or flashing back to the reference image?
  4. Table 3 evaluates audio-visual binding as a whole, leaving the individual contributions of the learnable audio marker and temporal RoPE alignment unclear. Could the authors explain their distinct roles, particularly what the marker contributes beyond providing an anchor for positional alignment?
Reviewer_p8h3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CineRef, a unified framework for cinematic reference-guided audio-video generation that supervises and maintains reference identity across the entire clip explicitly, rather than using references purely as passive conditioning. Implemented on the LTX-2.3 backbone through LoRA, it comprises three modules — Dual-Anchor Reference Alignment, Reference Trajectory Consistency, and Audio-Visual Reference Binding. On OpenS2V and the newly introduced CineSet benchmark, CineRef attains the strongest identity preservation together with lower inference cost than MiniMax-H3.

Critical strengths
  1. The identity-preservation findings are persuasive and well-controlled: on OpenS2V, CineRef reaches the best TotalScore with a wide FaceSim-Cur margin, and the same-backbone comparison against LTX-Ingredients credibly credits the improvement to the proposed reference modeling rather than the underlying base model.
  2. The "face-scale bias" diagnosis is a valuable, concrete observation: demonstrating that ArcFace identity supervision on aligned crops permits the model to reduce its loss by enlarging in-frame faces — and addressing this with a face-masked content anchor — is well-justified and supported qualitatively in Figure 5.
  3. Reference Trajectory Consistency is appropriately tailored to the cinematic scenario: by binding only high-variation observations to a shared identity anchor it addresses long-range cross-shot drift, and the ablation verifies its particular value for multi-shot generation.
Critical weaknesses
  1. Identity supervision contributes limited technical novelty. The ArcFace ID loss (Eq. 3) is characterized by the authors themselves as "a straightforward way" and is standard in face-swapping and ID-conditioned generation; pairing an identity loss with a content/reconstruction loss in a dual-anchor arrangement closely echoes established practice such as SimSwap's weak feature matching loss, yet this lineage goes undiscussed in the related work. The paper ought to identify which prior methods already employ ID supervision during training and delineate its true delta.
  2. Contribution 1 functions as a motivating thesis more than as a novel, verified finding. The principle that references should be modeled explicitly and strongly rather than concatenated passively is already the shared premise of much cited work (Phantom-Data, Saber, RefAlign, DreamID-Omni), and the claim is never isolated in a controlled study — it is assumed and absorbed into the overall method stack. It should be reframed as motivation or narrowed to the more specific, testable assertion that frame-level supervision fails under shot transitions while clip-level trajectory supervision succeeds.
  3. The audio-related claims are overstated relative to the strongest competitor: on CineSet, CineRef's SpkSim trails MiniMax-H3 in all four settings, Sync-C is frequently lower, and GmeScore (0.4918) falls short of LTX-Ingredients (0.5366), so "consistent improvements in visual-audio correspondence" hold only against weaker in-house baselines, with a possible identity-versus-semantics trade-off left unexamined.
  4. The method is tightly bound to face priors and assessed on a small, self-constructed benchmark. Because all supervision depends on ArcFace/RetinaFace, extension to non-human or full-body identity is unclear; meanwhile CineSet contains only 100 samples, the audio-video comparison covers just three methods, and no user study appears in the main text, raising concerns about selection bias and reliability.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.