- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Dasheng AudioGen, a unified text-to-audio framework that generates mixed audio scenes (speech + music + sound effects) using structured multi-view captions and a flow-matching DiT operating on the DashengTokenizer semantic-acoustic latent space. The authors claim four contributions: end-to-end audio scene generation, structured multi-view captions, generation with a semantic-acoustic representation, and a comprehensive evaluation pipeline.
Critical strengths
- Clear writing. The paper is well structured; figures, tables, and the method/experiment sections are easy to follow.
- Reasonable choice of representation. Adopting the high-dimensional semantic-acoustic latent space from DashengTokenizer as the generation target is a sensible direction, and the ablations show a clear advantage over low-dimensional VAE latents for mixed-scene generation.
- Thorough experimentation. The paper evaluates on standard benchmarks (AudioCaps, MusicCaps, LibriTTS) as well as the MECAT mixed-audio categories, with ablations, human evaluation, and an LLM-as-judge metric. Overall generation quality is competitive.
Critical weaknesses
W1. Insufficient coverage of prior work
The paper claims to be "the first non-autoregressive flow-matching DiT for joint speech / music / SFX generation", but diffusion-based joint speech + environmental sound has been an active line since 2023 — VoiceLDM [1], Audiobox [2], VoiceDiT [3], ControlAudio [4] (the last even supports timing control, which this submission does not). The "music" in the demos is mostly simple melodies and instrumental backgrounds, which prior TTA systems already produce under the "sound effects" label. This is a labeling distinction rather than a new capability, and the paper should situate itself within this line.
W2. Limited technical novelty
Structured multi-view captioning (Contribution 2) is prompt rewriting and is already standard. In audio, HoliDubber [5] decomposes captions into speaker profile / speech instructions / audio caption. In video, multi-dimensional structured captioning has been standard since 2024 (GameGen-X [6], HunyuanVideo [7], AniMatrix [8], OVI [9]). The six-field schema is a direct application of this pattern to audio.
Semantic-acoustic representation (Contribution 3) is the central contribution of a separate paper [10] and is used off the shelf, so it should not count as an independent contribution here.
The two "core challenges" framed in the introduction — lack of joint speech/music/SFX training data and insufficient capacity of low-dimensional VAE latents — have already been addressed in prior work (e.g., ACAVCaps [11] and DashengTokenizer [10]). Framing them as unresolved is misleading.
W3. Caption schema does not scale to realistic needs
The six-field design cannot express multi-speaker dialogue (alternating or overlapping turns) or temporal control (when a given event occurs), both of which are common in real audio content. This limits practical applicability.
References
- VoiceLDM: Text-to-Speech with Environmental Context. Lee et al., 2023. arXiv:2309.13664.
- Audiobox: Unified Audio Generation with Natural Language Prompts. Vyas et al., 2023. arXiv:2312.15821.
- VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis. Jung et al., 2024. arXiv:2412.19259.
- ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. Jiang et al., 2025. arXiv:2510.08878.
- HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis. Guan et al., 2026. arXiv:2606.09098.
- GameGen-X: Interactive Open-world Game Video Generation. Che et al., 2024. arXiv:2411.00769.
- HunyuanVideo: A Systematic Framework for Large Video Generative Models. Kong et al., 2024. arXiv:2412.03603.
- AniMatrix: An Anime Video Generation Model that Thinks in Art, Not Physics. 2026. arXiv:2605.03652.
- Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation. 2025. arXiv:2510.01284.
- DashengTokenizer: One Layer is Enough for Unified Audio Understanding and Generation. Dinkel et al., 2026. arXiv:2602.23765.
- ACAVCaps: Enabling Large-Scale Training for Fine-Grained and Diverse Audio Understanding. Niu et al., 2026. arXiv:2603.24038.
Questions for authors
Please see weakness.
7 comments
2. Structured Conditioning (W2)
We recognize prior structured conditioning in HoliDubber and the cited video models, and will clarify this context and our contribution. We design a conditioning interface for joint speech, music, and sound-effect generation, covering the global scene, speech attributes, transcript, music, sound effects, and room acoustics, and systematically test its effect on generation.
Appendix E tests field organization with equal information: the unstructured baseline keeps all field text, removes field markers, and joins the text with spaces. Both models were independently trained for 400K steps on the same 77,000 hours of data.
| Dataset | Metric | Unstructured (equal information) | Structured fields |
|---|---|---|---|
| AudioCaps | FAD ↓ | 3.30 | 3.89 |
| AudioCaps | CLAP ↑ | 35.3 | 37.5 |
| MusicCaps | FAD ↓ | 7.72 | 2.60 |
| MusicCaps | CLAP ↑ | 41.3 | 49.7 |
| LibriTTS | WER (%) ↓ | 55.28 | 17.14 |
| LibriTTS | UTMOSv2 ↑ | 3.035 | 3.160 |
| MECAT | FAD ↓ | 2.932 | 2.451 |
| MECAT | CLAP ↑ | 41.0 | 44.3 |
| MECAT speech subsets | WER (%) ↓ | 28.32 | 27.31 |
| MECAT speech subsets | UTMOSv2 ↑ | 2.863 | 2.976 |
Structured conditioning improves 9 of 10 metrics. With equal text information and training budgets, these results support separating semantic roles to help the model use descriptions, identify spoken content, and improve single-domain and mixed-scene generation. We will emphasize this task-specific design and empirical evidence, while reducing claims about the novelty of structured formatting itself.
3. New Evidence for Representation Choice (W2) — Part 1
We study which representations support joint generation from large-scale real mixed audio, and test the roles of latent capacity, semantic information, and acoustic information. ACAVCaps and DashengTokenizer provide the data and representation foundations, but available mixed data alone does not ensure that a conventional acoustic VAE can learn text–audio relations in complex scenes. We test this through training and ablations.
Strong single-domain results do not directly transfer to training on mixed data. Appendix C compares both representations trained on single-domain or mixed data. All results below are evaluated on the LibriTTS test set.
| Training data | VAE WER (%) ↓ | DashengTokenizer WER (%) ↓ | VAE UTMOSv2 ↑ | DashengTokenizer UTMOSv2 ↑ |
|---|---|---|---|---|
| LibriTTS (single domain) | 6.40 | 12.20 | 1.48 | 3.02 |
| ACAVCaps (mixed) | 32.90 | 10.77 | 1.52 | 3.12 |
VAE WER rises from 6.40% with speech-only training to 32.90% with mixed-data training. The semantic–acoustic representation reaches 10.77% with mixed-data training and retains higher UTMOSv2. An acoustic representation that works in one domain need not be equally suited to learning generation from mixed data; our experiments show the advantage of semantic–acoustic representations in this setting.
Thank you for the specific related-work suggestions and for recognizing our representation choice and experimental coverage. We add the following experiments and clarifications.
1. Related Work and Contribution Scope (W1)
We recognize the pioneering contributions of VoiceLDM, Audiobox, VoiceDiT, and ControlAudio to joint generation of intelligible speech and environmental sounds. We will expand this discussion and revise the “first” claim. Our core contributions are structured conditioning, a unified generation representation, and systematic evaluation for complex mixed scenes, rather than simply adding music as a category.
VoiceLDM and VoiceDiT focus on speech with background sounds; Audiobox explores unified speech and sound generation and editing. These works laid the foundation for unified audio modeling. ControlAudio also jointly generates speech, music, and sound effects, mainly using synthetic mixtures of clean speech and non-speech backgrounds. Dasheng AudioGen instead trains on about 77,000 hours of real mixed audio to learn the joint distribution of scene components. ControlAudio is not open source, preventing a direct comparison. These works also lack dedicated evaluation of complex speech–music–sound-effect mixtures.
Dasheng AudioGen combines a unified text interface with high-dimensional semantic–acoustic representations for real mixed scenes, with systematic evaluation across individual domains and component combinations: