- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SceneGen, a method for fine-grained temporal control in multi-shot audio-video generation, together with SceneGen-Data, a dataset of 194K annotated scenes. SceneGen-Data is built from Koala-36M, Panda-70M, and additional internet videos through an automated multi-stage pipeline. On the modeling side, SceneGen separates shot transitions from speech and sound events through two components, Shot-Aware RoPE and Event-Aware Attention. The model is built on LTX-2.3 and evaluated with video, audio, and multi-shot control metrics, together with a human evaluation.
Critical strengths
The paper addresses a relevant challenge in multi-shot audio-video generation. A shot boundary marks a visual discontinuity, but speech and sound events may continue across it, so audio timing requires control independent of shot boundaries. The paper makes this distinction explicit by modeling shot transitions separately from audio events. This is the main consideration in favor of the paper, since the problem is practically important for multi-shot generation.
Critical weaknesses
-
The experimental evidence does not clearly substantiate the central claim of fine-grained temporal control. The metrics are introduced only briefly in Section 3.1 and deferred to the appendix, and the main text does not identify or discuss those most relevant to this claim. The gains over LTX-2.3 are small, and no audiovisual examples are provided. This is consequential because the control ability itself is the paper's main contribution.
-
The contribution beyond existing components is limited and insufficiently analyzed. Shot-Aware RoPE closely follows MultiShotMaster, the ablations do not separate the two components or examine δ, and there is no analysis of why the design works. The dataset does not offset this, as its pipeline chains existing models that are themselves imperfect, so errors are likely to accumulate across stages, yet no verification or correction of the resulting annotations is reported. As a result, it remains unclear what the paper contributes beyond the base model and prior work. Addressing this would require substantial new analysis and data validation rather than clarification alone.
Questions for authors
Clarification:
- What does Shot-Aware RoPE change relative to MultiShotMaster (Section 2.2), and how sensitive are the results to δ?
- Which metrics directly reflect the claimed temporal control, and how does SceneGen compare with LTX-2.3 on them?
- What are the filtering thresholds and the character anchoring procedure used for SceneGen-Data, and what are the additional internet sources? Did the authors measure annotation quality at each stage, for example through human verification of a sample, and how are errors from earlier stages prevented from propagating to later ones?
Suggestion (not affecting my assessment): 4. Audiovisual examples showing event timing across cuts, and a description of the planned dataset release, including source licenses, would strengthen the paper.
5 comments
Weakness 2 & Question 3:
1. Filtering Thresholds, Character Anchoring, and Supplementary Sources Details are provided in Section 2.1, Figure 5, and Appendix B.2:
- Filtering Pipeline: We segment videos into 5 to 30 s scenes via TransNetV2, filter low visual and audio quality clips via DOVER++ and Audiobox Aesthetics, discard static clips via RAFT optical flow, remove scenes with over 5 people, strip persistent letterboxes, black frames, and heavy text overlays, inpaint residual subtitles and logos via RapidOCR and STTN, and remux the original audio.
- Character Anchoring and Sources: We detect faces and poses on sampled frames, merge cross-shot face clusters, and select reference crops by detection confidence, face size, and frontal visibility. Supplementary web videos cover film dialogue, vlogs and tutorials, public speaking, music performances, sports, cooking, and nature.
2. Preventing Error Propagation Instead of cascading text rewrites across models, we use single-stage evidence injection. Upstream detectors including TransNetV2, face clustering, WhisperX, and DiariZen only pass lightweight timestamps and identity anchors to Qwen3-Omni, which attends directly to raw video frames and full audio to globally verify event boundaries and speakers.
3. Human Verification of Final Annotations We conducted human verification on 100 randomly sampled scenes, added to Appendix B in the revision:
| Annotation Metric | Accuracy / Error |
|---|---|
| Shot Boundary Accuracy | 98.6% |
| Visual Description Accuracy | 94.7% |
| On-Screen Dialogue and Character ID Association | 93.1% |
| Single-Speaker Identification Accuracy | 94.7% |
| Overall Speaker Identification Accuracy | 92.5% |
| Speech Event Precision / Recall | 98.6% / 93.5% |
| Median Speech Onset Error | 0.05 s |
| Speech Onset / Offset Accuracy within 0.5 s | 93.2% / 95.0% |
Suggestion 4 & Ethics Review:
1. Demo Videos and Training Set Caption Demos Generated audio-visual demos and training set caption demos are available at https://anonymous.4open.science/r/Demo-A79D/.
2. Release Scope and Copyright Compliance
- Pipeline Code and Public Subsets: We will open-source the complete filtering and annotation code. Following the original licenses of Panda-70M and Koala-36M, we will make this part of data the same lisenses as Panda-70M and Koala-36M.
- Internal Web Data: Due to copyright restrictions, supplementary internal web videos and their annotations will not be released.
Dear Reviewer 9S3P,
We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the rebuttal discussion period draws to a close, we would greatly appreciate it if you could take a moment to review our rebuttal and share any further questions or concerns you may have.
Thank you once again for your valuable feedback and consideration.
Thank you for recognizing that our work addresses an important challenge in multi-shot audio-video generation and for acknowledging the practical value of modeling shot transitions separately from audio events. We address your specific questions and concerns below:
Weakness 1 & Question 2:
1. Metric Definitions and Discussion in the Main Paper All fine-grained temporal control metrics are defined, reported, and discussed in the main paper, while Appendix C.3 only covers implementation details:
2. Direct Comparison with LTX-2.3 The table below extracts the core temporal control metrics from Table 2. Because LTX-2.3 does not support explicit shot-cut timestamps, its shot-level metrics are N/A, so we also include multi-shot baselines HoloCine and MultiShotMaster:
Compared with LTX-2.3, SceneGen reduces STE by 93.5%, WER by 87.8%, and Desync by 26.3%, while outperforming HoloCine and MultiShotMaster on PF and CC. Audio-visual demos are available at https://anonymous.4open.science/r/Demo-A79D/.
Weakness 2 & Question 1:
1. Relation to MultiShotMaster and Core Contributions As stated in Sections 2.2 and 4, Shot-Aware RoPE directly adapts the discrete positional offset from MultiShotMaster. We treat Shot-Aware RoPE purely as an auxiliary component for controllable shot boundaries rather than a methodological innovation. Our core contributions lie in the fine-grained construction of the SceneGen-Data pipeline and the co-designed Event-Aware Attention mechanism.
2. Component-Wise Ablation in Table 3 Table 3 and Section 3.4 isolate both components under the same training budget against the SFT-only baseline:
3. Sensitivity Analysis on Δτ The table below reports results across Δτ values, where Δτ=0 s equals standard RoPE without shot offsets:
At Δτ=0 s, lacking shot isolation raises CTE to 3.67, whereas performance stays stable across Δτ=0.5 s and 1.0 s.