- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Soundwich, a training-free pipeline that grants existing text-to-video diffusion models the ability to generate layered audio. By composing existing techniques such as batch expansion and attention routing, and novel components such as carrier replay and scene broadcast, the system can generate multiple independently editable audio stems synchronized with a shared video. The model performs well against existing SOTA video generators (e.g., Gemini Omni, Minmax H3) on a small evaluation dataset. Soundwich shows better temporal controllability and source-level generation capability.
Critical strengths
- The problem and motivation are clearly defined. Single-stem audio is indeed a limitation for most current video generators. I especially appreciate the training-free approach, although some functions are only partially supported across models (e.g., entity routing is not available on Ovi 1.1).
- Great demo. I enjoyed watching the 10-minute video, which is a comprehensive demonstration of the model's capabilities. The audio quality and synchronization are good in the demo video.
- Novelty: 1) the gather-and-broadcast mechanism used to improve acoustic coherence is conceptually interesting; 2) using audio self-attention features from pretrained-model reference generations to control when individual sources are audible is very intuitive.
- The component-level ablations are convincing. All major design choices are ablated and backed with empirical results.
Critical weaknesses
- The benchmark is too small and insufficiently challenging to establish broad generalization. The main benchmark contains only 15 scenes, with 40 sound sources in total: 27 speech sources and 13 musical or other sound sources. I understand this is a training-free method and not a benchmark, but 15 is just too small; I suggest the authors add more scenes to make the experimental results more reliable. In Appendix E.4 about scene-level bootstrap, compared with Gemini-Omni, the 95% CI is [-2.8, 22.8], which makes the +8.1% on Window success not significant. I believe 50-100 scenes would be a reasonable number for such a task. A great demo video does not make up for insufficient evaluation. Currently, this is the biggest problem I see with this work. Please comment on any further plans to expand the scope of the experiments. I do not expect to see the full results for the expanded experiments, a reasonable plan and some initial results (if any) would help me make better decisions. I will increase my score based on the authors' response.
- Entity routing requires manual intervention. The authors stated that before Scene Integration, they need to visually inspect the mask generated by SAM and manually fix it by providing point prompts. This is a minor issue, but how many/what level of interventions do we need? Do most entities need it? Some statistics would help me understand whether the pipeline is labor-intensive or just needs to occasionally fix some corner cases. Also, did the authors consider using an LLM agent to automate this process? (Please don't conduct any experiments about the LLM agent, just some discussion is enough)
Questions for authors
Please see Weaknesses, W1 as Q1, W2 as Q2.
4 comments
I appreciate the authors' effort in extending the benchmark to 52 scenes. The new results of Soundwich look promising. I'll improve my score once the authors confirm they will finish the experiments on the extended evaluation data and put the results in the camera-ready version of the paper to replace the current main experiments.
I also thank the authors for exploring using an LLM agent to tackle the inaccurate mask issue. I'm glad it can help the pipeline to be more robust.
Dear reviewer, Thank you for your encouraging response. We confirm that we will complete the evaluation on 100 scenes and include the results for all compared methods in the camera-ready paper, replacing the current main experiments.
Thank you for the confirmation. As in my initial review, I very much appreciate the methodological design in this paper. Now, with enough empirical evidence (and more to come), I will increase my score to Weak Accept.
Response to Reviewer HcjL
Thank you for your detailed feedback. We agree that a larger benchmark would strengthen our results and would like to share our progress since submission.
Q1: Benchmark size and plan
Following your suggestion, we extended the benchmark further during the rebuttal period. We created 37 new scenes, so together with the original 15 scenes, the main benchmark now contains 52 scenes. We ran every method on them with the same protocol, which adds more than 650 new outputs. On the new scenes, Soundwich reaches 91.4% window success, compared with 54.2% for Gemini Omni, a difference of 37.2 points with a 95% confidence interval of [31.6, 42.8]. In the revision, we have merged the original 15 scenes and these 37 new scenes into the main table of the paper, so that it reports all 52 scenes. We have also added one output of every method for each new scene to the main comparison on our website.
We can extend the benchmark to 100 scenes if requested, although we expect similar trends. The main limitation is computational cost: our small university lab uses a single shared RTX 5090, and each scene requires three runs per method. Ovi alone takes up to 90 minutes per scene, while MiniMax H3 requires datacenter-class GPUs that are rarely available to us. Moreover, since no dataset exists for this task, we composed all scenes from scratch.
Q2: Mask intervention and LLM agents
Thank you for your helpful suggestion regarding LLM agents. As explained in our general response, SAM generates the initial masks, which an LLM agent reviews and corrects as needed. In the original submission, about two-thirds of the masks were usable without any correction, and only six cases required some adjustments. For the 37 new scenes evaluated during the rebuttal, we used a more recent LLM agent (Claude Opus 5.5), which successfully handled all masks without any human intervention. Thus, the pipeline is now almost fully automated.
We would be grateful for your opinion on whether these results address your concern, and we are happy to adjust it based on your feedback. Thank you again for your time and your thoughtful review.