- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a clear limitation of existing joint audio-video generators: they typically produce a single mixed soundtrack, limiting independent editing and spatial rendering of individual sound sources. It introduces SepGen, which extends a pretrained audio-video generator to jointly produce a video, its mixed soundtrack, and two text-specified source stems in a single sampling run. The method combines stem-specific LoRA adaptation, protected information flow that preserves the backbone’s video and mixed-audio outputs, caption routing, and cross-stem attention guidance. A two-stage training procedure and noise-dependent sampling schedules enable the same model to support both generation and source separation. Evaluation on generated scenes, externally generated videos, and real recordings, together with ablations, reference-stem evaluations, and listening studies, demonstrates improvements over the evaluated separation baselines.
Critical strengths
-
Clear motivation and a useful contribution. Generating two text-addressable stems alongside video and mixed audio addresses a practical limitation of existing audio-video generators and supports independent audio editing.
-
A coherent unified design. Protected information flow and stem-specific adaptation preserve the backbone’s outputs, while noise-dependent sampling enables generation and separation with the same weights.
-
Broad experimental validation. Evaluation spans generated and real scenes, combining automatic metrics, listening studies, and informative ablations and controls to support the reported improvements.
Critical weaknesses
-
Scalability beyond two sources remains unverified. The implemented layout and all experiments are restricted to two stems. It remains unclear how separation quality, cross-stem interference, and computational cost scale to scenes with more sources.
-
The listening studies have limited scene coverage. Each study includes only ten clips, limiting the generalizability of the perceptual results. The advantage on the generation study’s sounds subset is modest and varies across sources, with source-level majority votes yielding six wins, one tie, and five losses.
Questions for authors
Could you clarify whether extending it to three or more sources would require architectural changes or retraining, and whether any preliminary results demonstrate its scalability in both generation and separation?
4 comments
We thank Reviewer MfjX again for the review. As the discussion period closes soon, we would like to briefly follow up on our response.
We hope and believe that we have addressed all the questions raised by the reviewer. If the reviewer finds that indeed the reply and additions made to the paper address the concerns raised, we kindly ask the reviewer to consider raising the score. We are happy to clarify any remaining point before the discussion closes.
Thank you to the authors for the detailed response and additional experiments. I appreciate the efforts to clarify my concerns, particularly regarding scalability beyond two sources.
While the new results are encouraging, the performance degradation in three-source generation and the limited coverage of the listening studies still leave some concerns about the generalizability of the findings. Therefore, I will maintain my original rating of 2 (Weak Rejection).
We thank Reviewer MfjX for the response and for acknowledging the new results.
On three-source generation, the scores in Table S17 come from a checkpoint trained with two sources only and applied to three stems with no retraining. On speech the result is nearly unchanged (word error 0.13 against 0.11 with two stems), and every two-stem cascade baseline is at 0.81 or above (Table 1, three seeds). On sounds the Judge score moves from 3.82 to 3.61, which is still above FlowSep, the cascade baseline with the highest Judge score, on two stems at the same seed (3.49, Table S6). Training the generation stage with a third stem, for the same 3,000 updates, brings the three-stem Judge score to 3.85, on par with the two-stem result of 3.82.
On the listening studies, the automatic metrics on the full benchmarks (194 generated scenes, 149 Veo scenes and 172 real recordings) agree with the listeners and rank SepGen first (Tables 1, 2 and 3).
The preliminary three-source results provide evidence that, while our paper focuses on two stems, the method has the potential to extend to three stems as well.
Response to Reviewer MfjX
We thank the reviewer for the careful review and constructive feedback.
W1. “Scalability beyond two sources remains unverified.” “It remains unclear how separation quality, cross-stem interference, and computational cost scale to scenes with more sources.” In order to test how SepGen scales beyond two sources, we run our generation checkpoint with three stems as it is (the zero-shot checkpoint) and compare it with a checkpoint that we trained further with a third source (which we call the three-source checkpoint), in separation and in generation. These preliminary results are in the new Appendix H.
W2. “The listening studies have limited scene coverage. Each study includes only ten clips, limiting the generalizability of the perceptual results.” “The advantage on the generation study’s sounds subset is modest and varies across sources, with source-level majority votes yielding six wins, one tie, and five losses.” Our 95% intervals come from a bootstrap that resamples both the clips and the listeners, so their width already includes the uncertainty from having only ten clips.
In both studies, the overall interval for SepGen lies above the interval for the baseline (Table S3). On generated sounds, listeners prefer SepGen more often than the baseline (0.49 against 0.38). On generated speech, SepGen wins all seven sources (0.87 against 0.10).
The clips come from three benchmarks, including real recordings. By majority vote, 47 of 48 listeners in separation and 47 of 51 in generation prefer SepGen, so the preference does not come from a few listeners. On the full benchmarks (194 generated scenes, 149 Veo scenes and 172 real recordings), the automatic metrics also rank SepGen first (Tables 1, 2 and 3).
Q1. “Could you clarify whether extending it to three or more sources would require architectural changes or retraining, and whether any preliminary results demonstrate its scalability in both generation and separation?” No architectural change is needed. Each source adds one token region and one caption route, so our checkpoint runs with three stems as it is. Preliminary results show that no retraining is needed either. We run our generation checkpoint with three stems as it is (zero-shot transfer) and compare it with a checkpoint that we trained with a third source. With zero-shot transfer, SepGen separates three on-screen instruments, separates off-screen sounds into a third stem, and generates a third stem with background sound. Further training with a third source can help as it raises the swap margin on three on-screen instruments at a similar LSD but is not mandatory. The preliminary results are in the new Appendix H, in separation (Tables S15 and S16) and in generation (Table S17).