acceptodds
Under review as a conference paper at ICLR 2027

SceneGen: Generating Multi-Shot Scenes with Timed Speech and Sound

Abstract

Recent advances in joint audio-video generation have enabled realistic scenes with synchronized speech and sound. However, fine-grained temporal control remains challenging for open models, especially when camera transitions, visual actions, and audio events exhibit distinct temporal patterns within the same scene. Global conditioning cannot define local timing, while shot-level conditioning restricts events to camera boundaries, disrupting cross-shot continuity. To address this limitation, we propose \MethodName, a framework for fine-grained temporal conditioning that models shot transitions and audio events separately. Event-Aware Attention independently encodes complete speech and sound events, activating them based on their timeline intervals. Shot-Aware RoPE explicitly represents camera transitions through shared temporal position shifts across the audio and visual modalities. To provide supervision consistent with these mechanisms, we construct \DatasetName, containing approximately 194K multi-shot scenes with raw audio, linked character identities across views, shot-level visual annotations, and independently timed audio events. Together, these designs allow the model to follow shot-specific instructions while preserving events across camera cuts. Experiments on \BenchmarkName demonstrate improved speech timing, prompt following, and cross-shot character consistency, maintaining overall audiovisual quality. We will make our dataset and models publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fine-grained Control Generation, Audio-Video Multishot Generation, Video Generation Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9S3POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SceneGen, a method for fine-grained temporal control in multi-shot audio-video generation, together with SceneGen-Data, a dataset of 194K annotated scenes. SceneGen-Data is built from Koala-36M, Panda-70M, and additional internet videos through an automated multi-stage pipeline. On the modeling side, SceneGen separates shot transitions from speech and sound events through two components, Shot-Aware RoPE and Event-Aware Attention. The model is built on LTX-2.3 and evaluated with video, audio, and multi-shot control metrics, together with a human evaluation.

Critical strengths

The paper addresses a relevant challenge in multi-shot audio-video generation. A shot boundary marks a visual discontinuity, but speech and sound events may continue across it, so audio timing requires control independent of shot boundaries. The paper makes this distinction explicit by modeling shot transitions separately from audio events. This is the main consideration in favor of the paper, since the problem is practically important for multi-shot generation.

Critical weaknesses
  1. The experimental evidence does not clearly substantiate the central claim of fine-grained temporal control. The metrics are introduced only briefly in Section 3.1 and deferred to the appendix, and the main text does not identify or discuss those most relevant to this claim. The gains over LTX-2.3 are small, and no audiovisual examples are provided. This is consequential because the control ability itself is the paper's main contribution.

  2. The contribution beyond existing components is limited and insufficiently analyzed. Shot-Aware RoPE closely follows MultiShotMaster, the ablations do not separate the two components or examine δ, and there is no analysis of why the design works. The dataset does not offset this, as its pipeline chains existing models that are themselves imperfect, so errors are likely to accumulate across stages, yet no verification or correction of the resulting annotations is reported. As a result, it remains unclear what the paper contributes beyond the base model and prior work. Addressing this would require substantial new analysis and data validation rather than clarification alone.

Questions for authors

Clarification:

  1. What does Shot-Aware RoPE change relative to MultiShotMaster (Section 2.2), and how sensitive are the results to δ?
  2. Which metrics directly reflect the claimed temporal control, and how does SceneGen compare with LTX-2.3 on them?
  3. What are the filtering thresholds and the character anchoring procedure used for SceneGen-Data, and what are the additional internet sources? Did the authors measure annotation quality at each stage, for example through human verification of a sample, and how are errors from earlier stages prevented from propagating to later ones?

Suggestion (not affecting my assessment): 4. Audiovisual examples showing event timing across cuts, and a description of the planned dataset release, including source licenses, would strengthen the paper.

5 comments

Thank you for recognizing that our work addresses an important challenge in multi-shot audio-video generation and for acknowledging the practical value of modeling shot transitions separately from audio events. We address your specific questions and concerns below:

Weakness 1 & Question 2:

1. Metric Definitions and Discussion in the Main Paper All fine-grained temporal control metrics are defined, reported, and discussed in the main paper, while Appendix C.3 only covers implementation details:

  • Definitions: Section 3.1 defines Speech Boundary Error STE, Word Error Rate WER, Incorrect Utterance Rate Incorrect, Audio-Visual Desynchronization Desync, Cut Timing Error CTE, 5-point Prompt Following PF, and Cross-Shot Character Consistency CC.
  • Results and Analysis: Table 2 reports all quantitative results, which Section 3.2 analyzes under Fine-Grained Speech Control and Fine-Grained Multi-Shot Control.

2. Direct Comparison with LTX-2.3 The table below extracts the core temporal control metrics from Table 2. Because LTX-2.3 does not support explicit shot-cut timestamps, its shot-level metrics are N/A, so we also include multi-shot baselines HoloCine and MultiShotMaster:

MethodSTE ↓WER ↓Incorrect ↓Desync ↓CTE ↓PF ↑CC ↑
HoloCineN/AN/AN/AN/A1.744.024.08
MultiShotMasterN/AN/AN/AN/A2.314.093.69
LTX-2.3 Backbone3.526.4722.840.38N/AN/AN/A
SceneGen0.230.7914.320.281.974.194.31

Compared with LTX-2.3, SceneGen reduces STE by 93.5%, WER by 87.8%, and Desync by 26.3%, while outperforming HoloCine and MultiShotMaster on PF and CC. Audio-visual demos are available at https://anonymous.4open.science/r/Demo-A79D/.

Weakness 2 & Question 1:

1. Relation to MultiShotMaster and Core Contributions As stated in Sections 2.2 and 4, Shot-Aware RoPE directly adapts the discrete positional offset from MultiShotMaster. We treat Shot-Aware RoPE purely as an auxiliary component for controllable shot boundaries rather than a methodological innovation. Our core contributions lie in the fine-grained construction of the SceneGen-Data pipeline and the co-designed Event-Aware Attention mechanism.

2. Component-Wise Ablation in Table 3 Table 3 and Section 3.4 isolate both components under the same training budget against the SFT-only baseline:

  • Shot-Aware RoPE Only: Lowers CTE from 4.27 to 1.52 and improves CC from 3.79 to 4.17, but cannot control cross-shot audio timing, leaving STE at 3.26 and WER at 2.85.
  • Event-Aware Attention Only: Lowers STE from 3.62 to 0.36, WER from 1.79 to 0.68, and Desync from 0.43 to 0.31, while CTE remains at 3.67 without shot offsets.
  • Full Model vs. Joint Encoding: Combining both achieves balanced control with 1.97 CTE, 4.31 CC, 0.23 STE, 0.79 WER, and 0.28 Desync, outperforming Joint event encoding and verifying the necessity of decoupled modeling for independent visual cuts and audio events.

3. Sensitivity Analysis on Δτ\Delta\tau The table below reports results across Δτ\Delta\tau values, where Δτ=0 s\Delta\tau = 0\text{ s} equals standard RoPE without shot offsets:

Offset ConfigurationWER ↓STE ↓Desync ↓CTE ↓PF ↑CC ↑
Δτ=0 s\Delta\tau = 0\text{ s}0.680.360.313.674.264.21
Δτ=0.5 s\Delta\tau = 0.5\text{ s}0.790.230.281.974.194.31
Δτ=1.0 s\Delta\tau = 1.0\text{ s}0.700.250.262.054.324.29

At Δτ=0 s\Delta\tau = 0\text{ s}, lacking shot isolation raises CTE to 3.67, whereas performance stays stable across Δτ=0.5 s\Delta\tau = 0.5\text{ s} and 1.0 s1.0\text{ s}.

Weakness 2 & Question 3:

1. Filtering Thresholds, Character Anchoring, and Supplementary Sources Details are provided in Section 2.1, Figure 5, and Appendix B.2:

  • Filtering Pipeline: We segment videos into 5 to 30 s scenes via TransNetV2, filter low visual and audio quality clips via DOVER++ and Audiobox Aesthetics, discard static clips via RAFT optical flow, remove scenes with over 5 people, strip persistent letterboxes, black frames, and heavy text overlays, inpaint residual subtitles and logos via RapidOCR and STTN, and remux the original audio.
  • Character Anchoring and Sources: We detect faces and poses on sampled frames, merge cross-shot face clusters, and select reference crops by detection confidence, face size, and frontal visibility. Supplementary web videos cover film dialogue, vlogs and tutorials, public speaking, music performances, sports, cooking, and nature.

2. Preventing Error Propagation Instead of cascading text rewrites across models, we use single-stage evidence injection. Upstream detectors including TransNetV2, face clustering, WhisperX, and DiariZen only pass lightweight timestamps and identity anchors to Qwen3-Omni, which attends directly to raw video frames and full audio to globally verify event boundaries and speakers.

3. Human Verification of Final Annotations We conducted human verification on 100 randomly sampled scenes, added to Appendix B in the revision:

Annotation MetricAccuracy / Error
Shot Boundary Accuracy98.6%
Visual Description Accuracy94.7%
On-Screen Dialogue and Character ID Association93.1%
Single-Speaker Identification Accuracy94.7%
Overall Speaker Identification Accuracy92.5%
Speech Event Precision / Recall98.6% / 93.5%
Median Speech Onset Error0.05 s
Speech Onset / Offset Accuracy within 0.5 s93.2% / 95.0%

Suggestion 4 & Ethics Review:

1. Demo Videos and Training Set Caption Demos Generated audio-visual demos and training set caption demos are available at https://anonymous.4open.science/r/Demo-A79D/.

2. Release Scope and Copyright Compliance

  • Pipeline Code and Public Subsets: We will open-source the complete filtering and annotation code. Following the original licenses of Panda-70M and Koala-36M, we will make this part of data the same lisenses as Panda-70M and Koala-36M.
  • Internal Web Data: Due to copyright restrictions, supplementary internal web videos and their annotations will not be released.

Dear Reviewer 9S3P,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the rebuttal discussion period draws to a close, we would greatly appreciate it if you could take a moment to review our rebuttal and share any further questions or concerns you may have.

Thank you once again for your valuable feedback and consideration.

Reviewer_ErKAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SceneGen fine-tunes the LTX-2.3 audiovisual denoiser so that a camera cut and a line of speech can sit on different timelines. Shot-Aware RoPE adds 0.50.5s times the shot index to the temporal position of video and audio tokens, so the two sides of a cut are not treated as neighboring frames. Event-Aware Attention encodes each speech or sound event on its own and opens that text only inside its own interval, so one sentence can continue after the cut. Training uses about 194K annotated scenes. On SceneGen-Bench, speech-boundary error is 0.230.23s and shot-boundary error is 1.971.97 frames.

Critical strengths

The two timelines are separated, and Table 3 matches that split.

Shot-Aware RoPE alone moves shot-boundary error from 4.274.27 to 1.521.52. Event-Aware Attention alone moves speech-boundary error from 3.623.62 to 0.360.36 and WER from 1.791.79 to 0.680.68.

The full model holds both. This is the result that makes the design concrete.

Critical weaknesses
  1. There is no comparison with an agent-driven method, or with a fixed script written in advance. That setup first generates only the video with the shot cuts, and then adds the dubbing. Please add this comparison.

  2. Shot-Aware RoPE pushes the position ids after a cut off the continuous timeline the pretrained model uses. After the camera cuts, does the same person's identity still hold the way it does with the original continuous RoPE? Please answer this directly.

  3. The audiovisual backbones in the comparison are weak. Please compare with stronger models, such as Seedance 2.5 and MiniMax H3. This would let me judge how well the current best models handle this problem.

Questions for authors
  1. Please compare SceneGen with an agent-driven pipeline, or a fixed script written in advance, that first generates only the video with the shot cuts and then adds the dubbing.
  2. Please compare the original continuous RoPE with Shot-Aware RoPE: after a camera cut, does the person's identity still hold?
  3. Please compare with Seedance 2.5 and MiniMax H3.
2 comments

Thank you for your constructive feedback and for recognizing the complementary roles of Shot-Aware RoPE and Event-Aware Attention in our work. Below are our responses to your questions.

Q1: Comparison with a Two-Stage Pipeline

We construct a two-stage baseline using MultiShotMaster for multi-shot video generation followed by UNISON for post-hoc audio dubbing. Since the visual frames are identical to MultiShotMaster in Table 2, we report the newly added audio metrics below.

MethodCU ↑PQ ↑WER ↓Incorr. ↓
MultiShotMaster + UNISON4.925.863.2424.57
SceneGen5.566.770.7914.32

Compared to end-to-end joint generation, this two-stage agent pipeline increases error accumulation across modules and introduces additional system complexity. Visual timing and motion mismatches from the first stage propagate directly to the dubbing stage, resulting in higher speech errors and lower perceptual audio quality.

Q2: Cross-Shot Character Identity with Shot-Aware RoPE

Yes. Shot-Aware RoPE preserves cross-shot character identity after camera cuts.

Qualitative multi-shot videos generated by our method are available at https://anonymous.4open.science/r/Demo-A79D/, showing that recurring characters retain consistent appearance across viewpoint changes. Quantitatively, Table 3 reports cross-shot character consistency CC scores of 4.17 for SFT with Shot-Aware RoPE and 4.31 for the full SceneGen model, compared to 3.79 for the SFT baseline and 4.21 for SFT with Event-Aware Attention. These results confirm that Shot-Aware RoPE maintains character identity across cuts while reducing shot boundary error CTE from 4.27 to 1.52. Mechanistically, Shot-Aware RoPE shifts only the 1D temporal coordinate at shot boundaries while keeping spatial RoPE coordinates and global character conditioning unchanged across all shots.

Q3: Comparison with Stronger Baselines

MiniMax H3 is an advanced audiovisual generation model with strong synthesis capabilities. We evaluate MiniMax H3 under the same human evaluation protocol as Table 4.

ModelPF ↑Aesthetics ↑CC ↑Env. ↑Audio Quality ↑
MiniMax H34.084.404.363.844.02
SceneGen4.124.284.403.644.04

We acknowledge that a gap remains between SceneGen and MiniMax H3 in visual aesthetics and environmental sound generation, where MiniMax H3 benefits from stronger foundational synthesis capabilities. At the same time, SceneGen achieves higher scores on fine-grained prompt following PF and cross-shot character consistency CC. In terms of research focus, our work primarily aims to explore a structured multi-shot audiovisual data construction pipeline and investigate how decoupled temporal conditioning mechanisms can equip existing backbones with fine-grained control over independent visual and audio timelines.

Dear Reviewer ErKA,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the rebuttal discussion period draws to a close, we would greatly appreciate it if you could take a moment to review our rebuttal and share any further questions or concerns you may have.

Thank you once again for your valuable feedback and consideration.

Reviewer_2bcfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors propose a framework including a dataset and a model to improve audio-video generation that reflects fine-grained temporal control. The authors aggregate 194K multi-shot scenes with raw audio, linked character identities across views, shot-level visual annotations, and timed audio events. For modeling, the authors adopted the Hot-Aware RoPE method to accommodate camera transitions. Experiments on the proposed benchmark showed improved speech timing, prompt following, and cross-shot character consistency, etc. The experiment and human evaluation show favor for the proposed method.

Critical strengths

The notable strength of the paper is a full pipeline of work, starting from collecting SceneGen-Data to develop a model with event-aware attention and Shot-Aware RoPE. Thorough evaluation that demonstrates the superiority of the proposed model’s performance is also key strength.

Critical weaknesses

#Dataset Detail Currently, the annotation method is not detailed. For example, when I read through the annotation method section in Appendix B, it provides numeric information about the dataset, but does not fully explain how the whole dataset was annotated. Also, it is not clear how the authors chose videos on the web to be included in the dataset. Overall, it would also be interesting to see how authors could guarantee the quality of the collected and annotated dataset. I think the major weakness is that the dataset has not been shared with the reviewer, and it is hard to check the quality of the proposed dataset.

#Qualitative Results & Demo Video For audio-video generation, I think it is critical to provide a qualitative comparison with existing art by simply showing the videos. With the lack of a video, I am not confident in upholding the proposed method by just looking at the paper. If this paper were about image generation, it would not have been necessary. However, I strongly feel that the authors should have provided the full output in the video as supplementary. By looking at the numbers in Table 3, it is hard to figure out whether SceneGen really had a meaningful leap with the proposed method.

#Evaluation For human evaluation, I think this result should be reconsidered. The main reason is that the human evaluation should always be analyzed with the post-hoc statistical analysis. Without statistical analysis, it is not possible to claim superiority over others. Also, I am not sure what PF and CC metrics are. For human evaluation, a 5-point or 7-point Likert scale is usually used.

Questions for authors
  1. Provide a comparison with previous works by directly comparing audio-video output from your method and previous methods.
  2. How did you carry out the annotation? How can you guarantee the accuracy and performance of the annotation? Was filtering done systematically? If so, how do you gurantee the filtering performance?
  3. What were the criteria for collecting the videos on the web? I would like to hear how video collections were categorized, whether they cover all types of data.
  4. What are PF and CC metrics? How are they different from a conventional Likert scale?
  5. In terms of the model development, it seems the proposed work utilizes Shot-Aware RoPE and Event-Aware Attention without much modification. Could it be considered a technical contribution? If so, why?
  6. Please explicitly clarify in what areas the proposed system works better than previous works and which major problems the proposed system tries to solve.
6 comments

Thank you for recognizing the completeness of our pipeline, spanning SceneGen-Data construction and our model design, and for your constructive feedback. We address each of your concerns and questions below.

Weakness 1 & Questions 2, 3:

1. Web Video Collection and Domain Coverage, Question 3 As detailed in Section 2.1 and Appendix B, we source raw videos from Panda-70M, Koala-36M, and web platforms, obtaining 6,945 deduplicated videos and 339,823 candidate multi-shot scenes, which yield 194K final scenes after filtering. To ensure broad coverage, the dataset is balanced across seven domains: movie and TV dialogue, vlogs and tutorials, public speaking, music and performances, sports and dance, cooking, and nature, with 77.6% human-centric and 22.4% non-human scenes.

2. Systematic Filtering Pipeline, Question 2 We apply a multi-stage automated filtering pipeline:

  • Quality and Motion Filtering: DOVER++ filters visual quality, Audiobox Aesthetics filters acoustic quality, and RAFT optical flow removes static slideshows.
  • Artifact Removal: A pedestrian detector removes overcrowded scenes with more than five people at a 15.0% rejection rate. We also discard clips with persistent black borders, black frames, or heavy text overlays, and apply RapidOCR with STTN to detect and inpaint residual subtitles and logos before remuxing original audio.

3. Annotation Pipeline and Human Audit, Question 2 As shown in Figure 2, we extract annotations across three temporal granularities: cross-shot character IDs via face and pose clustering, shot-bounded visual descriptions, and continuous audio events via WhisperX, DiariZen, and Qwen3-Omni. To avoid error accumulation from cascading multiple vision-language models, we inject lightweight timestamps and character anchors into a unified Qwen3-Omni annotator that directly processes raw video and audio. A human audit on 100 random scenes confirms annotation reliability:

Annotation MetricAccuracy / Error
Shot Boundary Accuracy98.6%
Visual Description Accuracy94.7%
On-Screen Dialogue and Character ID Association93.1%
Single-Speaker Identification Accuracy94.7%
Overall Speaker Identification Accuracy92.5%
Speech Event Precision / Recall98.6% / 93.5%
Median Speech Onset Error0.05 s
Speech Onset / Offset Accuracy within 0.5 s93.2% / 95.0%

4. Dataset Sharing, Weakness 1 Representative annotated samples with raw video clips and full JSON annotations are now available at https://anonymous.4open.science/r/Demo-A79D/. Upon acceptance, we will release the full processing codebase and all redistributable scenes and annotations from Panda-70M and Koala-36M.


Weakness 2 & Questions 1, 6:

1. Qualitative Video Demonstrations, Question 1 We have uploaded qualitative video demonstrations across diverse scenarios to https://anonymous.4open.science/r/Demo-A79D/ so reviewers can directly inspect our audiovisual outputs.

2. Addressed Problems and Advantages, Question 6 Prior methods either rely on global prompts without precise temporal control or tightly bind all conditions to shot boundaries, abruptly cutting off ongoing speech or ambient audio across shot transitions. SceneGen decouples discrete shot switches from continuous audio timelines. In Table 2, SceneGen outperforms existing baselines across three key dimensions:

  • Speech Timing and Intelligibility: Over the LTX-2.3 backbone, SceneGen reduces Speech Timing Error STE by 93.5% from 3.52 to 0.23, lowers Word Error Rate WER from 6.47 to 0.79, and improves Desync from 0.38 to 0.28.
  • Multi-Shot Visual Control: SceneGen achieves the highest Prompt Following PF score of 4.19 and Cross-shot Character Consistency CC score of 4.31, surpassing HoloCine at 4.02 and 4.08 as well as MultiShotMaster at 4.09 and 3.69.
  • Audiovisual Quality: SceneGen achieves the top Text Alignment TA score of 23.44 while preserving perceptual audio quality with CU of 5.56 and PQ of 6.77.

Weakness 3 & Question 4: Human Evaluation Metrics

Addressing Question 4, PF stands for Prompt Following and CC stands for Cross-shot Character Consistency, both defined in Section 3.1. We will provide clearer definitions and detailed evaluation rubrics for both metrics in the revised Table 4 and Appendix C.3.


Question 5: Scope of Technical Contributions

We clarify our contribution boundaries as follows:

  • Shot-Aware RoPE: As explicitly stated in the main text, we follow prior work in adopting Shot-Aware RoPE. Because it is well suited for delineating shot boundaries, we include it purely as an implementation component to provide reliable shot-boundary control rather than claiming it as a primary methodological contribution.
  • Core Contributions: Our primary contributions are the SceneGen-Data curation and evidence-injected annotation pipeline and Event-Aware Attention in Section 2.3. Co-designed with our event-level captions, Event-Aware Attention injects explicit temporal event priors into attention layers to enable fine-grained audio control across shot transitions.

We will incorporate all clarifications, human audit statistics, and repository links into the revised manuscript.

Thanks for the response. How can we ensure that the given annotation metric is sufficiently compelling to validate the dataset's robustness? Are there any comparisons with prior work to determine whether these results are solid and sufficient?

For Q5, could you please elaborate more clearly? I understand that the "evidence-injected annotation pipeline and Event-Aware Attention" are new, but I am not sure whether they are important. So, please further clarify the impact of the proposed contribution on the ICLR community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.