acceptodds
Under review as a conference paper at ICLR 2027

Soundwich: Video Generation with Layered and Controllable Audio

Abstract

Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation.

Primary area:
generative models
Keywords:
audio-video generation, controllable generation, multi-track audio generation, multimodal generative models, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HcjLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes Soundwich, a training-free pipeline that grants existing text-to-video diffusion models the ability to generate layered audio. By composing existing techniques such as batch expansion and attention routing, and novel components such as carrier replay and scene broadcast, the system can generate multiple independently editable audio stems synchronized with a shared video. The model performs well against existing SOTA video generators (e.g., Gemini Omni, Minmax H3) on a small evaluation dataset. Soundwich shows better temporal controllability and source-level generation capability.

Critical strengths
  1. The problem and motivation are clearly defined. Single-stem audio is indeed a limitation for most current video generators. I especially appreciate the training-free approach, although some functions are only partially supported across models (e.g., entity routing is not available on Ovi 1.1).
  2. Great demo. I enjoyed watching the 10-minute video, which is a comprehensive demonstration of the model's capabilities. The audio quality and synchronization are good in the demo video.
  3. Novelty: 1) the gather-and-broadcast mechanism used to improve acoustic coherence is conceptually interesting; 2) using audio self-attention features from pretrained-model reference generations to control when individual sources are audible is very intuitive.
  4. The component-level ablations are convincing. All major design choices are ablated and backed with empirical results.
Critical weaknesses
  1. The benchmark is too small and insufficiently challenging to establish broad generalization. The main benchmark contains only 15 scenes, with 40 sound sources in total: 27 speech sources and 13 musical or other sound sources. I understand this is a training-free method and not a benchmark, but 15 is just too small; I suggest the authors add more scenes to make the experimental results more reliable. In Appendix E.4 about scene-level bootstrap, compared with Gemini-Omni, the 95% CI is [-2.8, 22.8], which makes the +8.1% on Window success not significant. I believe 50-100 scenes would be a reasonable number for such a task. A great demo video does not make up for insufficient evaluation. Currently, this is the biggest problem I see with this work. Please comment on any further plans to expand the scope of the experiments. I do not expect to see the full results for the expanded experiments, a reasonable plan and some initial results (if any) would help me make better decisions. I will increase my score based on the authors' response.
  2. Entity routing requires manual intervention. The authors stated that before Scene Integration, they need to visually inspect the mask generated by SAM and manually fix it by providing point prompts. This is a minor issue, but how many/what level of interventions do we need? Do most entities need it? Some statistics would help me understand whether the pipeline is labor-intensive or just needs to occasionally fix some corner cases. Also, did the authors consider using an LLM agent to automate this process? (Please don't conduct any experiments about the LLM agent, just some discussion is enough)
Questions for authors

Please see Weaknesses, W1 as Q1, W2 as Q2.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HcjL

Thank you for your detailed feedback. We agree that a larger benchmark would strengthen our results and would like to share our progress since submission.

Q1: Benchmark size and plan

Following your suggestion, we extended the benchmark further during the rebuttal period. We created 37 new scenes, so together with the original 15 scenes, the main benchmark now contains 52 scenes. We ran every method on them with the same protocol, which adds more than 650 new outputs. On the new scenes, Soundwich reaches 91.4% window success, compared with 54.2% for Gemini Omni, a difference of 37.2 points with a 95% confidence interval of [31.6, 42.8]. In the revision, we have merged the original 15 scenes and these 37 new scenes into the main table of the paper, so that it reports all 52 scenes. We have also added one output of every method for each new scene to the main comparison on our website.

Method (37 new scenes)Window WER ↓Window success ↑Full-clip sound ↑Order ↑Off-window ↓Assignment ↑
Naive batch expansion0.69317.895.271.161.073.2‾\underline{73.2}
Standard LTX-2.50.50125.595.290.338.3N/A
Ovi0.56918.581.086.845.6N/A
MiniMax H30.40726.295.294.434.4N/A
Gemini Omni 1.2 Flash (36 scenes)0.098‾\underline{0.098}54.2‾\underline{54.2}97.6‾\underline{97.6}96.7‾\underline{96.7}18.9‾\underline{18.9}N/A
Soundwich0.05091.4100.0100.01.487.9

We can extend the benchmark to 100 scenes if requested, although we expect similar trends. The main limitation is computational cost: our small university lab uses a single shared RTX 5090, and each scene requires three runs per method. Ovi alone takes up to 90 minutes per scene, while MiniMax H3 requires datacenter-class GPUs that are rarely available to us. Moreover, since no dataset exists for this task, we composed all scenes from scratch.

Q2: Mask intervention and LLM agents

Thank you for your helpful suggestion regarding LLM agents. As explained in our general response, SAM generates the initial masks, which an LLM agent reviews and corrects as needed. In the original submission, about two-thirds of the masks were usable without any correction, and only six cases required some adjustments. For the 37 new scenes evaluated during the rebuttal, we used a more recent LLM agent (Claude Opus 5.5), which successfully handled all masks without any human intervention. Thus, the pipeline is now almost fully automated.

We would be grateful for your opinion on whether these results address your concern, and we are happy to adjust it based on your feedback. Thank you again for your time and your thoughtful review.

I appreciate the authors' effort in extending the benchmark to 52 scenes. The new results of Soundwich look promising. I'll improve my score once the authors confirm they will finish the experiments on the extended evaluation data and put the results in the camera-ready version of the paper to replace the current main experiments.

I also thank the authors for exploring using an LLM agent to tackle the inaccurate mask issue. I'm glad it can help the pipeline to be more robust.

Dear reviewer, Thank you for your encouraging response. We confirm that we will complete the evaluation on 100 scenes and include the results for all compared methods in the camera-ready paper, replacing the current main experiments.

Thank you for the confirmation. As in my initial review, I very much appreciate the methodological design in this paper. Now, with enough empirical evidence (and more to come), I will increase my score to Weak Accept.

Reviewer_ZA8AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Soundwich is a training-free framework that adapts a pretrained joint audio-video model to generate a shared video with multiple independently editable audio stems. It combines source-specific guidance, cached activation and quiet carriers for temporal control, a scene stem for cross-source context sharing, and entity-guided attention routing. Experiments include generation comparisons, component ablations, human evaluations, and adaptations to additional backbones.

Critical strengths
  1. The paper addresses a practically meaningful limitation of joint audio-video generation: producing only a mixed soundtrack. Generating separate, controllable stems provides useful capabilities for remixing, retiming, and replacing individual sounds without retraining the backbone.
  2. The framework offers a coherent design for coordinating source separation, temporal control, and visual ownership. Component ablations provide evidence for carrier replay, quiet-reference features, scene broadcast, and entity routing.
Critical weaknesses
  1. The multi-stage pipeline may introduce substantial generation overhead. Reference construction, separate audio trajectories, segmentation, and audiovisual refinement add computational and operational complexity. Although carriers can be reused, the overall cost may limit practical use, particularly for new sound categories or scenes with more sources.
  2. The workflow depends on manual preparation and scene-specific choices. Carrier assignment, mask inspection and optional correction, source participation in scene sharing, and aggregation settings require decisions beyond specifying the desired content.
  3. Handling changing source visibility deserves further investigation. Audio-only sources are supported, but entity routing relies on masks initialized from the first frame. Robustness to sources entering the frame, becoming partially occluded, or reappearing is not systematically evaluated.
  4. Human evaluation covers only standard LTX-2.5, naive batch expansion, and Soundwich. Including Gemini Omni, MiniMax H3, and Ovi would provide a more complete assessment of naturalness and audiovisual consistency, particularly since these models are already included in the quantitative comparison.
  5. The carrier mechanism would benefit from further analysis. Existing ablations support its effectiveness, but the choices of injection location, RMS-based token selection, and category-matched references remain largely empirical. Their sensitivity and individual contributions are not yet clear.
Questions for authors
  1. Could the authors provide an end-to-end cost comparison with the original backbone, distinguishing first-use preparation from cached generation and including segmentation and refinement?
  2. How much manual intervention was required in the evaluation, particularly for mask correction and scene-specific configuration? How does performance change with uncorrected masks and fixed configuration rules?
  3. How does the method handle sources absent from the first frame or those whose visibility changes during generation? Could examples or evaluations clarify its robustness to occlusion and reappearance?
  4. Could the human evaluation include Gemini Omni, MiniMax H3, and Ovi for naturalness and audiovisual consistency?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZA8A

Thank you for your constructive review. We appreciate that you recognize the practical value of layered generation and the ability to remix, retime, and replace individual sounds without retraining the backbone. Your questions address important aspects of using Soundwich in practice, and we would like to address each of them below.

Q1: End-to-end cost

Soundwich has a modest computational cost. On a single RTX 5090, the original LTX-2.5 backbone takes about 1.5 minutes to generate a complete output, while Soundwich takes about two minutes for one stem and three minutes for four stems (Appendix F.4). Thus, increasing the number of stems by a factor of four increases the runtime by only about 70%. Moreover, Soundwich generates four stems in half the time required to generate them independently using the original backbone (3 minutes vs. 6 minutes).

First-use preparation is also light. Each sound category needs one reference generation to build its carrier. Because the carrier averages features over time, the reference can be short, and generating it takes about half a minute. After that, the carrier is cached and reused across scenes, so a new scene needs no new carriers unless it introduces a new category of sound. Segmentation adds about half a minute per output.

Q2: Manual intervention

For masks, we use the largely automatic pipeline described in the general response, and we include the manual review in the few remaining cases, because mask quality is important in the ownership of each stem.

We see the other main inputs as creative choices: what each source should sound like, and when it should play. Beyond these, all method settings are the same across scenes, and each source has only two remaining choices: its mix volume, and whether a continuous source such as music or ambience takes part in scene sharing.

Q3: Changing source visibility

SAM3.1 follows each source through the whole clip, so the masks are not tied to the first frame alone. When a source leaves the frame and comes back, the tracker picks it up again. For example, in Clockwork Airship, the main example on our website and in our demo video, the captain knocks on the door, opens it and walks into the room, and her mask follows her as she enters. A source that is absent at the start can be prompted on a later frame, since tracking runs in both directions.

Q4: Human evaluation scope

Following your suggestion, we ran an additional user study with 12 people that includes Gemini Omni, MiniMax H3 and Ovi. For each of 12 benchmark scenes, participants ranked the outputs by overall preference, considering timing, naturalness, quality, and how fully each video delivers the requested sounds. Soundwich received the best average rank (1.67 out of 4, compared with 1.87 for Gemini Omni, 2.75 for MiniMax H3 and 3.71 for Ovi) and was ranked first most often among all methods. We have included this study in the revision.

MethodMean rank ↓ (of 4)Ranked first ↑
Ovi3.712%
MiniMax H32.756%
Gemini Omni1.87‾\underline{1.87}43‾\underline{43}%
Soundwich1.6749%

W5: Motivation and effectiveness of the carrier design

Thanks for your question! We would like to clarify the intuition behind these design choices, as they were not made arbitrarily. The goal of the carrier mechanism is to provide the model with audio features relevant to the intended source. The use of category-matched references follows directly from this goal, as unrelated references would introduce irrelevant audio information. Similarly, RMS-based token selection identifies tokens with stronger audio activity, while the injection location allows us to introduce carrier information into the generation process. Our contribution is the use of carriers for source-specific audio generation, and we have demonstrated its effectiveness through our quantitative results and ablation studies. While alternative implementations are always possible, finding the optimal configuration for each of these design choices is not the focus of our work. We hope this clarifies the motivation behind our choices.

We hope these answers are helpful, and we would be glad to hear your thoughts. Thank you again for your constructive review and for the time you gave to our paper.

Thank you for the detailed response and extra results. The clarifications on computational cost, manual effort, and source tracking under changing visibility address most of my practical concerns. The new human evaluation against Gemini Omni, MiniMax H3, and Ovi also makes the experiments more convincing, and the added explanation makes the carrier design's motivation much clearer. Overall, most of my questions are resolved, so I will keep my Weak Accept score.

Reviewer_X29UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Soundwich, a training-free framework for layered audio-video generation. It expands the audio trajectory of a frozen joint generator into multiple source-specific trajectories while retaining a shared video, directly producing independently editable audio stems. Stem Formation uses per-stem CFG to suppress competing sounds and replays cached sounding and quiet attention features to control source activity, while an internal scene stem gathers shared context. Scene Integration re-noises and refines the initial outputs, broadcasts scene context, and routes cross-modal attention through entity masks to maintain the visual ownership of each sound. The evaluation includes a main comparison across 15 scenes and 40 sources, component ablations, human evaluations, additional scenes, and transfer to other backbones. The contribution is source-level control and editing interfaces during joint audio-video generation.

Critical strengths
  1. The framework provides a meaningfully different and useful output representation. Independent source stems allow sounds to be reordered, replaced, and remixed, followed by video refinement with the audio held fixed. Source-level control is applied directly within the joint generation process. Across 15 scenes, window success improves from 15.0% to 93.1% and off-window activity decreases from 69.7% to 1.7% relative to naive expansion using the same backbone (Table 1). These improvements and direct editability are the main reasons supporting acceptance. The commercial baselines produce a mixed soundtrack as their output interface. Sound quality requires an independent comparison alongside the evaluation of source-level control.

  2. The component design is matched by corresponding evaluations. The carrier ablation supports temporal control. CFG improves source assignment even when sources overlap completely and have no timing constraints. This provides evidence for source separation under fully concurrent activity. Human evaluations of scene broadcast and entity routing examine context and ownership, respectively. The interval for the source-assignment difference in the main comparison crosses zero. Visual ownership remains unreliable when the method is transferred to Ovi. Reporting these uncertainty estimates and transfer limitations improves the transparency of the evidence.

Critical weaknesses
  1. The evidence for source separation remains affected by selection and measurement. The main comparison measures source assignment using CLAP description matching. The amount of residual competing sound within each stem remains unquantified. The source-specific CFG ablation also selects cases with audible leakage. The results support mitigating leakage in cases where it is audible. The average benefit across randomly selected scenes remains to be established. Mixed-audio baselines additionally require SAM Audio extraction, which the authors' audit confirms can remove audible targets or retain non-target activity. The common-subset analysis and audit in the appendix provide important checks on these errors. Generation, extraction, and detection errors remain incompletely separated in the evaluation. More direct, blinded listening evaluations of target retention and competing content within each stem are needed to strengthen the general conclusion of cleaner separation.

  2. The dependence of entity routing on mask reliability and its workflow cost remain unquantified. The workflow allows users to inspect entity masks and correct them manually. The frequency of correction and the resulting ownership and naturalness without correction remain unreported. Appendix C.1 explicitly states that prompts, carrier assignments, volumes, and participation in scene sharing were fixed before the comparison across three seeds. Source-level manual inputs are also consistent with the interactive editing task itself. The remaining questions are how much of the routing improvement depends on additional mask correction, and how reliably it succeeds with small entities, occlusion, and complex interactions. These issues directly affect reproducibility and practical editing costs. H3 requires a modified implementation, and the transfer of visual ownership to Ovi remains unreliable. These results also indicate that the stability of the complete mechanism across backbones needs to be qualified.

Questions for authors
  1. Appendix C.1 states that source-level configurations were fixed before the comparison across three seeds. How many samples in the main comparison required manual point prompts to correct masks, at which stage were these corrections made, and what was the typical workload? This would affect the assessment of reproduction requirements and practical editing costs.

  2. Could you conduct a blinded evaluation of target retention and competing sounds in individual stems on concurrent-source scenes not selected for leakage? Please compare these results against the variant without source-specific CFG. Do the existing six leakage cases represent all tested outputs or a selected subset? This would directly test the scope of the separation conclusions.

  3. On the existing paired samples, with the established source configurations held fixed, how much do ownership and naturalness change between uncorrected and manually corrected masks? Could you report the proportions of routing failures and successful corrections for samples with small or occluded entities? This would directly test the stability of the benefits from entity routing.

2 comments
AuthorsOpenReview ↗

Response to Reviewer X29U

Thank you very much for your thorough review, and for asking such specific questions. We were also glad to read that you find the stem representation meaningfully different and useful. We take your concerns seriously and answer them in turn below.

Q1: Mask corrections in the main comparison

Please refer to our general response for details of the full process. We do not claim mask generation as a contribution of our work. In the original submission, 83 of 120 stem masks were used directly from SAM, while the agent corrected the remaining masks in a single review round before Scene Integration. Only about six cases required manual intervention. In our rebuttal experiments, using a stronger agent (Claude Opus 5.5) eliminated the need for manual mask corrections entirely.

Q2: Separation on scenes not selected for leakage

Source-specific guidance has two components: an individual prompt for each stem and a negative prompt that excludes competing sounds. The individual prompts are the primary mechanism for stem separation; without them, separation does not occur. Since removing them would be a trivial comparison, our ablation retains the individual prompts and removes only the negative prompts to isolate their effect.

As you correctly noted, our cases were specifically selected for audible leakage. Since negative prompts primarily target sound leakage, their effect is expected to be small in randomly selected scenes where little or no leakage occurs. We therefore focused on challenging cases where their contribution can be meaningfully evaluated. We acknowledge that this targeted evaluation does not establish their average benefit across arbitrary scenes, but believe it better isolates their intended role. We have acknowledged this limitation and better clarified the role of each component of source-specific guidance in the revised paper.

Q3: Uncorrected versus corrected masks

As explained in our general response, Soundwich assumes that entity masks are provided as input. Our main comparison therefore uses reviewed masks to evaluate our method independently of segmentation errors. However, mask quality affects both ownership and naturalness: incorrect masks can route sounds to the wrong visual entities and introduce leakage through incorrect V2A routing.

Our experiments show that routing is less reliable for small or heavily occluded entities, as they may cover very few video tokens or even less than one token (see Appendix F.5). This limitation is partly inherited from the underlying model, which itself struggles to generate and animate such objects. We have discussed this limitation in the paper.

W1: SAM Audio on mixed baselines

We agree that for mixed audio outputs, it is difficult to separate extraction errors from generation errors. We explicitly measure leakage in our generated stems using the CLAP margin metric, which measures the CLAP difference between the correct source and the highest competing source.

The main concern is therefore whether the leakage observed in the baselines comes from the generation model or SAM Audio. To address this, our comparison with naive batch expansion is particularly important, since both methods generate stems directly without any extraction. This comparison demonstrates the effectiveness of our method in reducing leakage independently of SAM Audio. In addition, we evaluate speech directly on the mixed audio without extraction (Appendix D) to avoid this issue in speech evaluation.

Finally, the difficulty of reliably extracting individual sounds from mixed audio, particularly when multiple sources produce similar or overlapping sounds, is itself one of our main motivations for generating stems separately in Soundwich.

We would be grateful for your thoughts on these points, and we are happy to discuss them further. Thank you again for your time and your careful review.

Thank you for the author's response. After reading the author's reply, I have decided to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thank you very much for your careful and thoughtful reviews. We appreciate the time you spent on our paper, and your feedback has been very helpful. We also appreciate that the reviewers have found our paper to address a practically meaningful limitation of existing joint audio-video generation models [ZA8A, HcjL], introduce a meaningfully different and useful output representation [X29U], and present a coherent and novel framework for source-level audio control [ZA8A, HcjL]. Below we respond to the concerns you share, and we address the remaining points in the individual replies. We have uploaded a revised paper with the changes marked in blue.

Entity masks

There was a common question regarding the masks used for entity routing [HcjL, X29U, ZA8A]. Soundwich takes the masks of visible sound sources as inputs, alongside text prompts and timelines. These masks can be generated by any segmentation method, as Soundwich only uses them to route the generated audio to the corresponding sources. Therefore, our contribution is the audio routing mechanism, not the segmentation itself.

The paper's phrase "manual mask correction" suggests more human effort than it really takes. In our experiments, we automated this step:

  1. SAM3.1 drafts each mask from a text description of the source
  2. An LLM agent then inspects the mask overlays and corrects a mask where needed by giving SAM a new prompt or one or two points.

As a result, preparing the masks requires minimal human effort and is not labor-intensive [HcjL]. In the paper’s benchmark, about two-thirds of the masks were directly usable from SAM, while the remaining masks were corrected by the agent (GPT-5.6 Sol). Manual intervention was needed in only about six cases out of the 45 benchmark outputs. For the 37 new scenes in this rebuttal, a newer agent (Claude Opus 5.5) reviewed every mask, and no manual intervention was needed at all. Furthermore, since our method is independent of the segmentation model, stronger segmentation methods can easily replace SAM as they become available, further reducing the need for manual intervention. We have clarified these details in the revision (Appendix C.3).

Gemini Omni

We appreciate the reviewers' questions regarding Gemini Omni [ZA8A, HcjL]. Omni is a closed-source commercial model, so we cannot control its settings or inspect how it works. Also, we cannot apply our method on top of it and compare it with the vanilla model for a fair comparison. Still, we include it as a reference, since it is one of the strongest available systems. In the paper, we collected its outputs using the authors’ free-trial quotas, which were limited and reset daily. As a result, we could collect only one output per scene, and we agree that the difference in window success across 15 scenes is not statistically significant [HcjL].

For this rebuttal, we evaluated the latest paid version of Gemini Omni (1.2 Flash) on 37 new scenes, generating three outputs per scene. On these scenes, Soundwich reaches ~91% window success, compared with ~54% for Omni, and the difference is statistically significant. We also collected two more outputs for each of the original 15 scenes, so the table in the new revision uses three Omni outputs for every scene. Since the service is continuously updated, we cannot verify the exact underlying model behind each output. Therefore, we report the model name and collection dates with these results in the revision (Appendix C.2).

Method (37 new scenes)Window WER ↓Window success ↑Full-clip sound ↑Order ↑Off-window ↓
Gemini Omni 1.2 Flash (36 scenes)0.09854.297.696.718.9
Soundwich0.05091.4100.0100.01.4

We also ran an additional user study that includes Gemini Omni [ZA8A]. In this study, 12 participants ranked the outputs of four methods on 12 benchmark scenes by overall preference. Soundwich received a better average rank than Gemini Omni and was ranked first more often. The full results are in our reply to Reviewer ZA8A.

MethodMean rank ↓ (of 4)Ranked first ↑
Gemini Omni1.8743%
Soundwich1.6749%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.