acceptodds
Under review as a conference paper at ICLR 2027

SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

Abstract

A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets will be released upon publication.

Primary area:
generative models
Keywords:
joint audio-video generation, audio source separation, language-queried audio separation, diffusion models, multi-source audio, spatial audio

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MfjXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a clear limitation of existing joint audio-video generators: they typically produce a single mixed soundtrack, limiting independent editing and spatial rendering of individual sound sources. It introduces SepGen, which extends a pretrained audio-video generator to jointly produce a video, its mixed soundtrack, and two text-specified source stems in a single sampling run. The method combines stem-specific LoRA adaptation, protected information flow that preserves the backbone’s video and mixed-audio outputs, caption routing, and cross-stem attention guidance. A two-stage training procedure and noise-dependent sampling schedules enable the same model to support both generation and source separation. Evaluation on generated scenes, externally generated videos, and real recordings, together with ablations, reference-stem evaluations, and listening studies, demonstrates improvements over the evaluated separation baselines.

Critical strengths
  1. Clear motivation and a useful contribution. Generating two text-addressable stems alongside video and mixed audio addresses a practical limitation of existing audio-video generators and supports independent audio editing.

  2. A coherent unified design. Protected information flow and stem-specific adaptation preserve the backbone’s outputs, while noise-dependent sampling enables generation and separation with the same weights.

  3. Broad experimental validation. Evaluation spans generated and real scenes, combining automatic metrics, listening studies, and informative ablations and controls to support the reported improvements.

Critical weaknesses
  1. Scalability beyond two sources remains unverified. The implemented layout and all experiments are restricted to two stems. It remains unclear how separation quality, cross-stem interference, and computational cost scale to scenes with more sources.

  2. The listening studies have limited scene coverage. Each study includes only ten clips, limiting the generalizability of the perceptual results. The advantage on the generation study’s sounds subset is modest and varies across sources, with source-level majority votes yielding six wins, one tie, and five losses.

Questions for authors

Could you clarify whether extending it to three or more sources would require architectural changes or retraining, and whether any preliminary results demonstrate its scalability in both generation and separation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MfjX

We thank the reviewer for the careful review and constructive feedback.

  • W1. “Scalability beyond two sources remains unverified.” “It remains unclear how separation quality, cross-stem interference, and computational cost scale to scenes with more sources.” In order to test how SepGen scales beyond two sources, we run our generation checkpoint with three stems as it is (the zero-shot checkpoint) and compare it with a checkpoint that we trained further with a third source (which we call the three-source checkpoint), in separation and in generation. These preliminary results are in the new Appendix H.

    • Separation quality. With three on-screen instruments, the zero-shot and three-source checkpoints have the highest Judge scores, and on LSD to the true stems they are second only to AudioSep (Table S15). When off-screen noise is added as a third source, we achieve lower LSD with three stems than with two stems and than every baseline (Table S16).
    • Cross-stem interference. With three on-screen instruments, the zero-shot and three-source checkpoints both have higher swap margins than every baseline (Table S15). The swap margin measures how much more a stem matches its own caption than the captions of the other stems. With off-screen noise, the noise moves to the third stem, so less of it remains in the on-screen stems (Table S16). In generation, the on-screen stems still match their own captions above the others, with a lower swap margin than with two stems (Table S17).
    • Computational cost. The generation time is almost the same for two and three stems, since the audio tokens are few compared to the video tokens (Section 4.1).
  • W2. “The listening studies have limited scene coverage. Each study includes only ten clips, limiting the generalizability of the perceptual results.” “The advantage on the generation study’s sounds subset is modest and varies across sources, with source-level majority votes yielding six wins, one tie, and five losses.” Our 95% intervals come from a bootstrap that resamples both the clips and the listeners, so their width already includes the uncertainty from having only ten clips.

    In both studies, the overall interval for SepGen lies above the interval for the baseline (Table S3). On generated sounds, listeners prefer SepGen more often than the baseline (0.49 against 0.38). On generated speech, SepGen wins all seven sources (0.87 against 0.10).

    The clips come from three benchmarks, including real recordings. By majority vote, 47 of 48 listeners in separation and 47 of 51 in generation prefer SepGen, so the preference does not come from a few listeners. On the full benchmarks (194 generated scenes, 149 Veo scenes and 172 real recordings), the automatic metrics also rank SepGen first (Tables 1, 2 and 3).

  • Q1. “Could you clarify whether extending it to three or more sources would require architectural changes or retraining, and whether any preliminary results demonstrate its scalability in both generation and separation?” No architectural change is needed. Each source adds one token region and one caption route, so our checkpoint runs with three stems as it is. Preliminary results show that no retraining is needed either. We run our generation checkpoint with three stems as it is (zero-shot transfer) and compare it with a checkpoint that we trained with a third source. With zero-shot transfer, SepGen separates three on-screen instruments, separates off-screen sounds into a third stem, and generates a third stem with background sound. Further training with a third source can help as it raises the swap margin on three on-screen instruments at a similar LSD but is not mandatory. The preliminary results are in the new Appendix H, in separation (Tables S15 and S16) and in generation (Table S17).

We thank Reviewer MfjX again for the review. As the discussion period closes soon, we would like to briefly follow up on our response.

We hope and believe that we have addressed all the questions raised by the reviewer. If the reviewer finds that indeed the reply and additions made to the paper address the concerns raised, we kindly ask the reviewer to consider raising the score. We are happy to clarify any remaining point before the discussion closes.

Thank you to the authors for the detailed response and additional experiments. I appreciate the efforts to clarify my concerns, particularly regarding scalability beyond two sources.

While the new results are encouraging, the performance degradation in three-source generation and the limited coverage of the listening studies still leave some concerns about the generalizability of the findings. Therefore, I will maintain my original rating of 2 (Weak Rejection).

We thank Reviewer MfjX for the response and for acknowledging the new results.

On three-source generation, the scores in Table S17 come from a checkpoint trained with two sources only and applied to three stems with no retraining. On speech the result is nearly unchanged (word error 0.13 against 0.11 with two stems), and every two-stem cascade baseline is at 0.81 or above (Table 1, three seeds). On sounds the Judge score moves from 3.82 to 3.61, which is still above FlowSep, the cascade baseline with the highest Judge score, on two stems at the same seed (3.49, Table S6). Training the generation stage with a third stem, for the same 3,000 updates, brings the three-stem Judge score to 3.85, on par with the two-stem result of 3.82.

On the listening studies, the automatic metrics on the full benchmarks (194 generated scenes, 149 Veo scenes and 172 real recordings) agree with the listeners and rank SepGen first (Tables 1, 2 and 3).

The preliminary three-source results provide evidence that, while our paper focuses on two stems, the method has the potential to extend to three stems as well.

Reviewer_ZJHqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method that extends a model generating sound together with video so that it simultaneously generates the audio-mix for the whole scene and the multiple stems that compose it. The method introduces latents for the stems into an existing joint audio-video generator, and then generates the video, the audio-mix, and the stems jointly using a flow-matching framework. The attention and LoRA within the model are designed so that the stem features do not affect the video and audio-mix features, and so that each stem reflects only its corresponding caption. As a result, the method can generate stems faithful to their captions while preserving the original video and audio-mix outputs. Two further components suppress leakage between stems: aligned rotary positions, which align the stems in time, and Cross-Stem Attention Guidance, which uses the sibling caption as the negative branch. In addition, by providing the (video and) audio-mix without noise, the same model can perform source separation. Training comprises two stages. It starts with the separation task and then briefly continues on noisy audio-mixes, so a single checkpoint supports both generation and separation. Experiments show that the method generates higher-quality stems than cascade baselines that perform generation and separation sequentially. It also outperforms existing methods on the separation task, and a user study confirms the same trend.

Critical strengths
  • The proposed method achieves stem generation by adding a lightweight adapter to a frozen backbone and training only that adapter, which makes it practically feasible and easy to use.
    • The attention and LoRA designs that keep the video and audio-mix latents unaffected are also well motivated. The ability to add new functionality without degrading the outputs of an existing generator is an important advantage of this method.
  • In the experiments, the proposed model performs well in both generation and separation tasks, even against powerful models such as SAM Audio. The examples on the demo page also clearly illustrate the superiority of the proposed model.
Critical weaknesses
  • In the abstract and introduction, the authors describe the motivation of this work from the perspective of rendering 4D scenes. However, the task addressed in this paper is mainly the joint generation of the audio-mix and the stems, and the spatial placement and rendering of sound sources are not evaluated. Therefore, there is some gap between the motivation and the scope of the actual experiments.
    • In the ablation study in Table 4, replacing the video with blank frames, an unrelated clip, or no video barely changes separation performance. This suggests that the video information does not contribute substantially to stem generation or separation. As the authors acknowledge, the stems retain the reverberation and distance cues of the original scene, so they cannot be used directly for rendering from a new viewpoint.
    • Because of this gap between the stated motivation and the actual contributions, the scope of the claims should be reframed around audio generation and separation.
  • The audio in the generation benchmark and the LTX-2.3 benchmark is produced by models of the same family as the backbone, which may give the proposed method an advantage in terms of domain. Given the results on the Veo benchmark and on real recordings, I do not expect this to overturn the conclusions, but a reference-based evaluation on a standard benchmark would further support the claims.
Questions for authors
  • The estimated clean audio-mix in Eq. (9) depends on the velocity predicted at the same timestep, but the paper does not specify how it is computed at each timestep (for example, whether an additional forward pass is required or the estimate from the previous step is reused). Could the authors elaborate on this and report the additional computational cost at generation time (if any) compared with the setting where the stems attend only to the noisy audio-mix (Table S7, "never")?

  • Two questions regarding the evaluation datasets:

    • (1) Why is the method not evaluated on reference-based benchmarks commonly used in source separation research, such as synthetic mixtures from MUSIC or VGGSound? Some of these datasets are also used for training, but would it be possible to report a reference-based evaluation (envelope correlation, LSD) as in Appendix F while taking care to avoid overlap with the training data?
    • (2) Why was only a subset of SAM-Audio-bench used?
  • [Suggestions for future work] Judging from Table 4, the contribution of the video information is quite limited in stem generation or separation. Related to this, it would be interesting to investigate the following points:

    • (1) Is the ability of the placement readout to localize sources coming from the source-box attention bias adopted in the training (Appendix A)? Is localization still obtained when the model is trained without this bias?
    • (2) Can the video information be leveraged for separation if object region information is provided explicitly, as in SAM Audio?
    • (3) Can the proposed method also be applied to text-to-audio models that do not handle video?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ZJHq

We thank the reviewer for the feedback.

  • W1. “The scope of the claims should be reframed around audio generation and separation.” We revised the presentation accordingly. The abstract now starts with multi-stem generation and separation. The introduction (Section 1) and the caption of Figure 1 give rerendering a 4D scene as one example application of per-source waveforms, and the conclusion (Section 5) keeps 4D rendering as future work.
  • W1 (Table 4). “The video information does not contribute substantially to stem generation or separation.” “The stems retain the reverberation and distance cues of the original scene.” In generation, SepGen makes the video, the audio-mix and the stems in one run, so the stems belong to the generated scene. It is true that separation does not rely on the video, so SepGen also separates recordings that have none (Table 4). As for reverberation, as the limitations section (Section 5) states, the stems keep the reverberation and distance cues of the scene they come from, so a spatial renderer must account for them before it places each source at a new viewpoint. We recognize this limitation.
  • W2. “A reference-based evaluation on a standard benchmark would further support the claims.” We added a reference-stem evaluation on 50 MUSIC-21 and 50 VGGSound mixes from videos absent from training (Table S13). SepGen is first on the 129 held-out scenes, second to AudioSep on MUSIC-21 and third on VGGSound, where these metrics mainly reward keeping the audio-mix (as reported in the general response, Reference stems). On the domain question, Veo and the real recordings of SAM-Audio-bench come from outside the backbone family, and SepGen has the best results on both (Tables 2 and 3).
  • Q1. “Whether an additional forward pass is required or the estimate from the previous step is reused”, and “the additional computational cost at generation time (if any) compared with the setting where the stems attend only to the noisy audio-mix (Table S7, ‘never’)”. The estimate is the prediction of the current step. It comes from the regular pass, so computing it needs no added pass. Below σ_g, one extra pass predicts the stems from this estimate and an additional guidance pass uses the captions of the other stems as negative guidance for CSAG. Both passes reuse the video features of the regular pass. The cost is approximately 4% more time per clip than “never”, for both two and three stems. We measured the generation time on A6000 and added it to Section 4.1. We have also added a clarification of this mechanism in Section 3.
  • Q2a. “Would it be possible to report a reference-based evaluation (envelope correlation, LSD) as in Appendix F while taking care to avoid overlap with the training data?” Yes. We added 50 MUSIC-21 and 50 VGGSound mixes from videos absent from training to Table S13, next to the 129 held-out scenes of Appendix F. Results and more information are presented in the general response (Reference stems).
  • Q2b. “Why was only a subset of SAM-Audio-bench used?” We use the three text subsets that describe the target: instruments, sound effects, and speakers (172 recordings). The other two subsets use general labels such as “music playing” and “speech”. A general label does not identify a specific source when a clip has many instruments or many speakers. We have now also scored these two subsets at seed 0, and the camera-ready version will report all five subsets over five seeds. SepGen keeps the highest Judge score and swap margins on this extended benchmark version as well (Appendix B).

Suggestions for future work.

  • S1. “Is the ability of the placement readout to localize sources coming from the source-box attention bias? Is localization still obtained when the model is trained without this bias?” Yes, localization is still obtained, the ability does not come from the source box attention. The box bias operates only in training. To measure this, we trained a checkpoint without the bias. On 64 clips, its azimuth error is 3.14 degrees, against 3.08 degrees with the bias. We added this clarification to Appendix G.
  • S2. “Can the video information be leveraged for separation if object region information is provided explicitly, as in SAM Audio?” We think so. In training, a soft bias already restricts the attention of each stem to its source box (Appendix A), and giving this region at inference would extend it. We have not experimented with it yet.
  • S3. “Can the proposed method also be applied to text-to-audio models that do not handle video?” We believe so. Separation does not rely on the video (Table 4), so the stem layout and Estimated Separation could apply to a text-to-audio backbone. We have not tested it.

Thanks for the detailed response and additional experiments.

Regarding W2:

In the added reference-based evaluation on MUSIC-21 and VGGSound (Table S13), the separation mode falls behind existing methods such as AudioSep. The response attributes this to these metrics mainly rewarding keeping the audio-mix. However, on VGGSound, the envelope correlation of AudioSep exceeds that of the audio-mix copied into both stems, so I do not think the nature of the metrics alone explains this result. As the authors acknowledge in Appendix F, most of the error of SepGen comes from sound it leaves out of its own source.

Taken together with the large margins of SepGen over the baselines on the caption-based metrics, this result suggests that the stems may be pulled toward sounds that are typical of their captions, rather than toward the components actually contained in the audio-mix. Table S16 is also consistent with this interpretation: on the held-out scenes with added background noise, the Judge score (4.11) and J-swap (0.97) of SepGen with three stems are on par with or higher than those of the true stems (4.10 and 0.91). This relates to the concern raised by Reviewer RyPY (W1) that the caption-based metrics reward a stem that sounds like its caption rather than one that is actually contained in the audio-mix.

From Appendix A, I understand that two kinds of text guidance are applied to the stems: CFG on the audio side (scale 7.0) and CSAG (λ=2). These may make the influence of the captions too strong. In that case, the large gaps on the caption-based metrics do not necessarily indicate gaps in how faithfully the stems are separated from the audio-mix.

Additional questions

  1. On the 50 mixes each of MUSIC-21 and VGGSound, could the authors show how the following two types of metrics change when the CFG scale and the CSAG λ for the stems are varied (including turning CSAG off)? These experiments require only inference.

    • Reference-based metrics: envelope correlation, LSD, and the decomposition of LSD into left-out and added sound used in Appendix F
    • Caption-based metrics: Judge and J-swap

    If weakening the guidance improves the reference-based metrics, please discuss the trade-off between the two types of metrics and the recommended setting.

  2. On the 129 held-out scenes, MUSIC-21, and VGGSound, could the authors report the Judge, J-swap, and C-swap of the true stems alongside those of SepGen and the baselines? Please also explain how the authors interpret the result in Table S16 that the caption-based metrics of SepGen are on par with or higher than those of the true stems.

1/2

We thank the reviewer for raising this point and for proposing the two experiments. This discussion deepens the paper and we believe it makes it substantially stronger. The two proposed experiments are reported in a new Appendix I (Figure S4 and Table S18), and the trade-off they show is discussed there and in the conclusion of the paper.

The main contribution of SepGen is multi-stem generation. To our knowledge, this is the first framework that turns a pretrained joint audio-video generator into a multi-stem generator that can simultaneously generate the video, its soundtrack, and individual stems in a single run. Separation is a complementary mode of this model. In generation the caption defines the source, so a close match to the caption is the goal. This is analogous to the perception side of the perception-distortion trade-off, an established research topic in image restoration and compression (Blau and Michaeli, 2018; Blau et al., 2018; Blau and Michaeli, 2019; Freirich et al., 2021). The reviewer is right that the same trade-off shows in separation. There the source already exists in the audio-mix, a generative separator can favor the sound its caption describes over the sound in the audio-mix, and the strength of the guidance controls how far it does. In audio, ZeroSep (Huang et al., 2025) also reports that the guidance strength affects separation with a text-guided audio diffusion model. We see a wide field for follow-up work in carrying this line of research to audio fidelity, in particular how this curve looks for speech vs. general audio.

We agree with the reviewer that on VGGSound SepGen trails AudioSep and FlowSep (Section 4.1), and most of its error is sound left out of each source. So the gap is not an issue caused by the metric alone (Appendix F). We have revised the claim in Appendix F to refer solely to LSD. Yet, we stress again that the main contribution of our method is the generative part and the separation is a complementary mode of it that we show can also be useful and competitive with leading approaches.

Reviewer_RyPYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SepGen, which extends a pretrained joint audio-video generator. In addition to the video and the audio-mix, it also outputs one waveform per captioned source. The stems are appended to the audio stream as extra token channels with aligned rotary positions and trained with LoRA. The audio-mix is protected by an attention mask and by turning the LoRA off on mix tokens, so the backbone's video and soundtrack stay unchanged. Each stem only attends to its own caption, and Cross-Stem Attention Guidance uses the sibling caption as a negative branch. The same model supports two modes, selected by the noise level of the audio-mix: 1) separation, where the observed mix is kept clean, and 2) generation, where the mix is denoised jointly and the stems read its clean estimate in later steps. The model is trained on about 4.8k two-source examples. It is evaluated on its own generated scenes, on scenes rendered by Veo 3.1 Lite and LTX-2.3, and on real recordings from SAM-Audio-bench, against language-queried separators. SepGen outperforms these baselines on caption-based metrics and in a user study.

Critical strengths
  • The idea is practical and the design is clean. Getting per-source stems from a joint audio-video generator in a single run, without changing what the backbone produces is useful. Using the noise level of the audio-mix to switch between generation and separation with one checkpoint is a simple and elegant choice. Training is also lightweight with only LoRA, about 4.8k examples, and a single GPU.

  • The evaluation is designed with care in several places. The swap margins and the degenerate controls (mix returned twice, silent stem) guard against trivial solutions. The VAE round-trip and band-limiting controls, and the best-of-eight comparison with SAM Audio, address the most obvious confounders. The results are consistent across generated, cross-generator, and real benchmarks, and the user study agrees with them.

Critical weaknesses
  • My main concern is that the evaluation does not directly measure how faithfully the stems are extracted, and the strongest results seem to rely on information that the baselines cannot use. The stems are re-synthesized by the audio VAE, so SI-SDR is not used. The main benchmarks also have no reference stems, so all reported metrics compare each stem with its caption. These metrics reward a stem that sounds like its caption, which is not the same as a stem that is actually contained in the mix, especially for a generative separator. The only reference-based evaluation (Appendix F) uses held-out scenes from SepGen's own training families. For speech, the captions include the exact spoken line, and WER is computed against that same line. So the large WER gap mainly reflects transcript conditioning. Once the lines are removed, SepGen's WER rises to 0.50, some stems become silent, and SepFormer reaches a similar WER (0.53). I think two things are needed to support the claim that SepGen is a better separator: evaluation on standard separation benchmarks with reference stems, and the no-transcript setting as the main speech comparison.

  • My second concern is the gap between the motivation and what is actually shown. The paper is motivated by rerendering 4D audio-visual scenes from new viewpoints, but there is no spatial rendering experiment: Figure 1b is an illustration, and the placement readout in Appendix G is only qualitative. The stems also keep the original reverberation, which, as the paper notes, a renderer would have to deal with. The layout is fixed at two sources. Table 4 shows that replacing the video with blank frames, an unrelated clip, or no video does not change the separation results, so in separation mode the model essentially behaves like an audio-only, text-queried separator. As a consequence, two sources of the same category cannot be told apart from the visual side, and same-instrument duets indeed get much worse LSD (11.7 vs. 6.3 dB). The technical changes are a fairly direct adaptation (extra stem tokens, an attention mask, LoRA gating, NAG with the sibling caption), similar in spirit to MSDM, MGE-LDM, and MMAudioSep. The contribution would be much stronger if the spatial use case, or a real benefit from the video, were demonstrated.

Questions for authors
  • Since the stems are re-synthesized and the main metrics are caption-based, how do you check that a stem does not contain plausible content that is absent from the mix? The donor test in Table S12 covers speech. Is there a similar check for sounds and music?

  • Could you report the no-transcript speech results on the generation and LTX-2.3 benchmarks as well, together with SepFormer/MossFormer2, and break them down for same-gender speaker pairs?

  • How would the layout extend to more than two sources, or to a variable number of sources? Would this require retraining for each NN?

  • Positional captions with the box bias are used in training. At inference, do positional captions help in the same-instrument case, where semantic captions cannot distinguish the two stems?

4 comments
AuthorsOpenReview ↗

Response to Reviewer RyPY (1/2)

1/2

We thank the reviewer for the constructive feedback.

  • W1. “I think two things are needed to support the claim that SepGen is a better separator: evaluation on standard separation benchmarks with reference stems, and the no-transcript setting as the main speech comparison.”
    • Reference stems. We added MUSIC-21 and VGGSound to the reference-stem evaluation of Table S13 in Appendix F. The general response describes the three sets and their results.

    • Speech without transcripts. To further test whether SepGen separates speech better than SepFormer and MossFormer2 with no transcripts in the captions, we removed every quoted line from the captions of all methods, across all three speech benchmarks. SepFormer and MossFormer2 have no support for captions and they return two unlabeled outputs, so each output has to be paired with one of the two captions while SepGen returns the stems coupled to their captions. To evaluate this, we propose two pairing protocols:

      • The protocol already presented in the paper (Table S11a) transcribes the two outputs and picks the pairing whose transcripts best match the quoted lines. It pairs the outputs of SepGen without the lines in the same way. This protocol uses the quoted lines only during the pairing process. Under this protocol, SepGen has the lowest word error rate on all three benchmarks.
      • Following the suggestion listed in the review, we introduce an additional 'no-transcript' CLAP protocol (Table S11b - upper block) which uses no transcript lines at all. It picks the pairing of the two outputs with the two captions that has the highest total CLAP audio-text similarity. We apply it to the outputs of every method, SepGen included. SepGen then has the lowest word error rate on all three benchmarks.

      We also report SepGen and the text-queried baselines without the CLAP assignment mechanism, since each output is already coupled with the caption it was produced from. In this case, the word error of SepGen is even lower, with 0.50, 0.50, and 0.49 on the generation, Veo and LTX-2.3 benchmarks, against 0.79, 0.69, and 0.78 for SAM large, the best baseline in that block (Table S11b, lower block).

    • The review compares SepGen’s 0.50 on Veo with SepFormer’s 0.53. These two numbers come from different protocols. The 0.50 is SepGen with its stems as they are and no lines in any step (Table 2). The 0.53 is SepFormer with its outputs paired by the quoted lines (Table S11a). With the transcript protocol for both, SepGen has 0.32 against 0.53 (Table S11a). With no lines for either and the same CLAP pairing, SepGen has 0.54 against 0.73 (Table S11b). SepGen keeps its lead over the speech separators without the quoted lines. The line-free setting is part of our main results, for every text-conditioned method in Table 2 (second value in each speech cell).

AuthorsOpenReview ↗

Response to Reviewer RyPY (2/2)

2/2

  • W2. “The contribution would be much stronger if the spatial use case, or a real benefit from the video, were demonstrated.”
    • Spatial use case. The project page and the supplementary video demonstrate the spatial use case. SepGen separates a clip and places each stem based on location achieved by the attention of the stem to the video. We have added a preliminary metric for this placement in Appendix G. On 64 clips, the azimuth error is 3.08 degrees against true source trajectories.
    • Benefit from the video. In the 4D use case, the location of each stem source comes from the video. The attention from each stem to the video tracks its source in the image, with no extra training or detector (Appendix G). In generation, SepGen makes the video, the audio-mix and the stems in one run, so the stems match the generated video.
    • Prior work. MSDM and MGE-LDM are audio-only, and MMAudioSep extracts one source for each pass. We know of no other method that generates a video and captioned stems in one run.
    • More sources and same-instrument duets. We answer these in Q3 and Q4.
  • Q1. “How do you check that a stem does not contain plausible content that is absent from the mix? The donor test in Table S12 covers speech. Is there a similar check for sounds and music?” We check that the stems add up to the audio-mix, so content absent from the mix would show as a mismatch. On the 194 clips of the generation benchmark, the sum of the stems agrees with the audio-mix, and no baseline agrees better on envelope correlation or LSD (Table S14). For sounds and music, we repeated the donor test on the 37 sounds and music clips of the LTX-2.3 set. The test keeps the captions and replaces the observed audio-mix with the audio-mix of a different clip, so a stem that only follows its caption would keep its score. In this case both CLAP and Judge scores drop with respect to the original audio-mix (Table S12). The stems therefore follow the audio-mix.
  • Q2. “Could you report the no-transcript speech results on the generation and LTX-2.3 benchmarks as well, together with SepFormer/MossFormer2, and break them down for same-gender speaker pairs?” We added Table S11b, which covers all three benchmarks with both separators, the text-queried baselines of Table 2, and a split for same-gender pairs. Same-gender pairs are hard for every method, and on the generation and Veo benchmarks the same-gender word error is 0.64 or more for all of them. This is where the caption helps, since it describes each speaker beyond the sound of the voice. SepGen has the most distinct voices on same-gender pairs on all three benchmarks, and the lowest same-gender word error on Veo and LTX-2.3.
  • Q3. “How would the layout extend to more than two sources, or to a variable number of sources? Would this require retraining for each K?” Each source adds one token region and one caption route, so a checkpoint trained with two stems can be extended to more sources. Preliminary results show that this also works with no retraining (Appendix H). Further training with a third source can help as it raises the swap margin on three on-screen instruments at a similar LSD but is not mandatory. The general response (More than two sources) describes the three settings and their results.
  • Q4. “At inference, do positional captions help in the same-instrument case, where semantic captions cannot distinguish the two stems?” Not really. Positional captions make the two captions different, which helps with achieving distinct stems but the stems do not follow the stated positions. When we mirror the video, each stem keeps the same player in 24 of the 26 duets (Appendix F).
Reviewer_RyPYOpenReview ↗

Official Comment by Reviewer RyPY

Thank you for the thoughtful response and for the additional experimental results.

Most of my concerns have been addressed, so I will maintain my positive rating for this paper.

We thank Reviewer RyPY for the constructive feedback and for the support.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers) (1/2)

1/2

We thank the reviewers for their feedback. Red text in the revised PDF shows all changes and additions to the paper and the appendix.

  • Speech without transcripts (RyPY). To further test whether the speech results of SepGen depend on the quoted lines in the captions, we removed all quoted lines from the captions. SepGen keeps the lowest word error rate and the most distinct voices on the three benchmarks, against SepFormer, MossFormer2 and the text-queried baselines (Table S11b). We added this comparison to Appendix E, and Section 4.3 of the paper summarizes it.
  • More than two sources (MfjX, RyPY). To measure whether SepGen handles more than two sources without architectural changes or retraining, we test our generation checkpoint with three stems (zero-shot transfer). We compare it with the baselines and with a checkpoint that we trained further with a third source (the three-source checkpoint). The results show that the zero-shot checkpoint separates and generates a third source as it is. These preliminary results are in the new Appendix H.
    • Three instruments on screen (Table S15). We separate 30 held-out MUSIC-21 clips where each shows three different instruments and caption each stem with one instrument, to test whether each stem faithfully extracts these instruments. Swap margin measures how much more a stem matches its own caption than the other captions. The zero-shot and three-source checkpoints have the highest Judge scores and swap margins. On log-spectral distance (LSD) to the true stems, both are at the top, second only to AudioSep. Further training with a third source can help as it raises the swap margin at a similar LSD but is not mandatory.
    • Off-screen noise as the third source (Table S16). To test whether a third stem can hold off-screen background sounds and noises, we add a noise recording to each of our 129 held-out scenes and caption the third stem as background sounds. With three stems, the on-screen stems have a lower LSD to their true stems than with two stems and than every baseline, so the noise moves to the third stem. The zero-shot and three-source checkpoints reach a similar LSD.
    • Generation with a third noise stem (Table S17). To test whether SepGen can generate a third stem with background sounds, we add the same background-noise caption as a third stem to every prompt of the generation benchmark. The on-screen stems still match their own captions above the others. The three stems together sum closer to the generated audio-mix than the on-screen stems alone, so the third stem captures part of the generated sound.
AuthorsOpenReview ↗

General response (to all reviewers) (2/2)

2/2

  • Reference stems (RyPY, ZJHq). To extend our comparison against true reference stems, we added MUSIC-21 and VGGSound to Table S13 of Appendix F, next to the 129 held-out scenes already in the paper. Section 4.1 summarizes the results. Each audio-mix sums two recordings, so the true stem of each source is known. We measure envelope correlation (whether a stem follows the loudness of its true source over time) and log-spectral distance (LSD, the distance to the spectrum of the true source). We compare SepGen with AudioSep, FlowSep and SAM Audio, each with one candidate per stem.
    • Held-out scenes (129). These are instrument duets, dialogues and composites from the datasets we train on, held out from training. SepGen is closest to the true stems on both metrics, also against SAM Audio with the best of eight candidates.

    • MUSIC-21 (50 mixes of two instrument videos absent from training). SepGen is second only to AudioSep. They are the only two methods with one candidate per stem that score above the audio-mix copied into both stems on both metrics. SAM Audio passes SepGen only when it picks the best of eight candidates per stem.

    • VGGSound (50 mixes of two everyday-sound videos absent from training). AudioSep is first, FlowSep second and SepGen third, ahead of both SAM Audio models. VGGSound recordings are broadband and sound for the whole clip, so the two sources overlap. On such data, LSD penalizes removing inherent sounds much more than keeping a sibling sound. The audio-mix copied into both stems, which separates nothing, has the lowest LSD, and most of the error of SepGen is sound it leaves out (Appendix F).

      We therefore also rely on the swap margins, on which a copied audio-mix scores zero and SepGen leads every baseline (Tables 2 and 3), and on the user study, where 47 of 48 listeners prefer SepGen over SAM Audio.

  • Scope (ZJHq, RyPY). We have reframed some of the presentation towards the general task of multi-stem audio-video generation and separation, with 4D scenes as the motivation. The abstract now starts with multi-stem generation and separation. The introduction (Section 1) and the caption of Figure 1 now present rerendering a 4D scene as one example application. The conclusion (Section 5) keeps 4D rendering as future work.
  • Other clarifications. We explain the cost of Estimated Separation (Section 4.1 in the paper), the test against the audio-mix of another clip (Table S12 in Appendix E), positional captions (Appendix F), and listener counts (Appendix C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.