acceptodds
Under review as a conference paper at ICLR 2027

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

Abstract

Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks.

Primary area:
generative models
Keywords:
Audio Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wn15OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dasheng AudioGen, a unified text-to-audio framework that generates mixed audio scenes (speech + music + sound effects) using structured multi-view captions and a flow-matching DiT operating on the DashengTokenizer semantic-acoustic latent space. The authors claim four contributions: end-to-end audio scene generation, structured multi-view captions, generation with a semantic-acoustic representation, and a comprehensive evaluation pipeline.

Critical strengths
  • Clear writing. The paper is well structured; figures, tables, and the method/experiment sections are easy to follow.
  • Reasonable choice of representation. Adopting the high-dimensional semantic-acoustic latent space from DashengTokenizer as the generation target is a sensible direction, and the ablations show a clear advantage over low-dimensional VAE latents for mixed-scene generation.
  • Thorough experimentation. The paper evaluates on standard benchmarks (AudioCaps, MusicCaps, LibriTTS) as well as the MECAT mixed-audio categories, with ablations, human evaluation, and an LLM-as-judge metric. Overall generation quality is competitive.
Critical weaknesses

W1. Insufficient coverage of prior work

The paper claims to be "the first non-autoregressive flow-matching DiT for joint speech / music / SFX generation", but diffusion-based joint speech + environmental sound has been an active line since 2023 — VoiceLDM [1], Audiobox [2], VoiceDiT [3], ControlAudio [4] (the last even supports timing control, which this submission does not). The "music" in the demos is mostly simple melodies and instrumental backgrounds, which prior TTA systems already produce under the "sound effects" label. This is a labeling distinction rather than a new capability, and the paper should situate itself within this line.

W2. Limited technical novelty

Structured multi-view captioning (Contribution 2) is prompt rewriting and is already standard. In audio, HoliDubber [5] decomposes captions into speaker profile / speech instructions / audio caption. In video, multi-dimensional structured captioning has been standard since 2024 (GameGen-X [6], HunyuanVideo [7], AniMatrix [8], OVI [9]). The six-field schema is a direct application of this pattern to audio.

Semantic-acoustic representation (Contribution 3) is the central contribution of a separate paper [10] and is used off the shelf, so it should not count as an independent contribution here.

The two "core challenges" framed in the introduction — lack of joint speech/music/SFX training data and insufficient capacity of low-dimensional VAE latents — have already been addressed in prior work (e.g., ACAVCaps [11] and DashengTokenizer [10]). Framing them as unresolved is misleading.

W3. Caption schema does not scale to realistic needs

The six-field design cannot express multi-speaker dialogue (alternating or overlapping turns) or temporal control (when a given event occurs), both of which are common in real audio content. This limits practical applicability.

References

  1. VoiceLDM: Text-to-Speech with Environmental Context. Lee et al., 2023. arXiv:2309.13664.
  2. Audiobox: Unified Audio Generation with Natural Language Prompts. Vyas et al., 2023. arXiv:2312.15821.
  3. VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis. Jung et al., 2024. arXiv:2412.19259.
  4. ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling. Jiang et al., 2025. arXiv:2510.08878.
  5. HoliDubber: Holistic Video Dubbing for Complex Acoustic Scenes via Text-Guided Audio Synthesis. Guan et al., 2026. arXiv:2606.09098.
  6. GameGen-X: Interactive Open-world Game Video Generation. Che et al., 2024. arXiv:2411.00769.
  7. HunyuanVideo: A Systematic Framework for Large Video Generative Models. Kong et al., 2024. arXiv:2412.03603.
  8. AniMatrix: An Anime Video Generation Model that Thinks in Art, Not Physics. 2026. arXiv:2605.03652.
  9. Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation. 2025. arXiv:2510.01284.
  10. DashengTokenizer: One Layer is Enough for Unified Audio Understanding and Generation. Dinkel et al., 2026. arXiv:2602.23765.
  11. ACAVCaps: Enabling Large-Scale Training for Fine-Grained and Diverse Audio Understanding. Niu et al., 2026. arXiv:2603.24038.
Questions for authors

Please see weakness.

7 comments

Thank you for the specific related-work suggestions and for recognizing our representation choice and experimental coverage. We add the following experiments and clarifications.

1. Related Work and Contribution Scope (W1)

We recognize the pioneering contributions of VoiceLDM, Audiobox, VoiceDiT, and ControlAudio to joint generation of intelligible speech and environmental sounds. We will expand this discussion and revise the “first” claim. Our core contributions are structured conditioning, a unified generation representation, and systematic evaluation for complex mixed scenes, rather than simply adding music as a category.

ModelBackboneSpeech + SFXSpeech + MusicMixed-Scene GenerationMixed-Scene EvalOpen Source
VoiceLDMDiffusion U-Net✓✓✗✗✓
AudioboxFlow Matching✓✗✗✗✗
VoiceDiTDiffusion DiT✓✓✗✗✗
ControlAudioDiffusion DiT✓✓✓✗✗
Dasheng AudioGenFlow Matching DiT✓✓✓✓✓

VoiceLDM and VoiceDiT focus on speech with background sounds; Audiobox explores unified speech and sound generation and editing. These works laid the foundation for unified audio modeling. ControlAudio also jointly generates speech, music, and sound effects, mainly using synthetic mixtures of clean speech and non-speech backgrounds. Dasheng AudioGen instead trains on about 77,000 hours of real mixed audio to learn the joint distribution of scene components. ControlAudio is not open source, preventing a direct comparison. These works also lack dedicated evaluation of complex speech–music–sound-effect mixtures.

Dasheng AudioGen combines a unified text interface with high-dimensional semantic–acoustic representations for real mixed scenes, with systematic evaluation across individual domains and component combinations:

  • Representation: 1280D semantic–acoustic latents replace low-dimensional acoustic VAE latents as the Flow Matching DiT target. Full, semantic-only, and acoustic-only ablations test this choice.
  • Training data: About 77,000 hours of real speech, music, and sound effects support large-scale pretraining on the joint distribution of mixed scenes.
  • Evaluation: AudioCaps, MusicCaps, and LibriTTS assess individual domains. MECAT covers music + sound effects, speech + sound effects, speech + music, and all three, assessing distribution quality, text and component alignment, speech intelligibility, and listening quality.
  • Open access: Model weights are public.

2. Structured Conditioning (W2)

We recognize prior structured conditioning in HoliDubber and the cited video models, and will clarify this context and our contribution. We design a conditioning interface for joint speech, music, and sound-effect generation, covering the global scene, speech attributes, transcript, music, sound effects, and room acoustics, and systematically test its effect on generation.

Appendix E tests field organization with equal information: the unstructured baseline keeps all field text, removes field markers, and joins the text with spaces. Both models were independently trained for 400K steps on the same 77,000 hours of data.

DatasetMetricUnstructured (equal information)Structured fields
AudioCapsFAD ↓3.303.89
AudioCapsCLAP ↑35.337.5
MusicCapsFAD ↓7.722.60
MusicCapsCLAP ↑41.349.7
LibriTTSWER (%) ↓55.2817.14
LibriTTSUTMOSv2 ↑3.0353.160
MECATFAD ↓2.9322.451
MECATCLAP ↑41.044.3
MECAT speech subsetsWER (%) ↓28.3227.31
MECAT speech subsetsUTMOSv2 ↑2.8632.976

Structured conditioning improves 9 of 10 metrics. With equal text information and training budgets, these results support separating semantic roles to help the model use descriptions, identify spoken content, and improve single-domain and mixed-scene generation. We will emphasize this task-specific design and empirical evidence, while reducing claims about the novelty of structured formatting itself.

3. New Evidence for Representation Choice (W2) — Part 1

We study which representations support joint generation from large-scale real mixed audio, and test the roles of latent capacity, semantic information, and acoustic information. ACAVCaps and DashengTokenizer provide the data and representation foundations, but available mixed data alone does not ensure that a conventional acoustic VAE can learn text–audio relations in complex scenes. We test this through training and ablations.

Strong single-domain results do not directly transfer to training on mixed data. Appendix C compares both representations trained on single-domain or mixed data. All results below are evaluated on the LibriTTS test set.

Training dataVAE WER (%) ↓DashengTokenizer WER (%) ↓VAE UTMOSv2 ↑DashengTokenizer UTMOSv2 ↑
LibriTTS (single domain)6.4012.201.483.02
ACAVCaps (mixed)32.9010.771.523.12

VAE WER rises from 6.40% with speech-only training to 32.90% with mixed-data training. The semantic–acoustic representation reaches 10.77% with mixed-data training and retains higher UTMOSv2. An acoustic representation that works in one domain need not be equally suited to learning generation from mixed data; our experiments show the advantage of semantic–acoustic representations in this setting.

Reviewer_JaMyOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dasheng AudioGen, a unified framework for generating complex audio scenes containing speech, music, and sound effects. The model achieves promising results in mixed-audio generation, particularly in terms of overall audio quality and coherence

Critical strengths
  1. The paper presents a simple and effective framework for end-to-end mixed-audio generation, without requiring separate generation models for different audio components.
  2. The paper provides extensive evaluations and ablation studies, including human evaluation and comparisons between structured and unstructured captions.
Critical weaknesses
  1. One of the main differences between the proposed system and conventional AudioLDM-style approaches is the use of a semantic-acoustic representation instead of a conventional acoustic VAE. Although the authors conduct ablation studies comparing the VAE and DashengTokenizer, the two representations have substantially different latent dimensions (128D vs. 1280D). Therefore, it is difficult to determine whether the improvements are mainly due to the semantic information or the increased latent capacity.

  2. Different audio conditions are concatenated into a single text sequence and processed through the same text encoder and cross-attention mechanism. Although structured captions improve generation quality, the paper does not clearly demonstrate whether the model can independently control different audio components. For example, it remains unclear whether changing the music condition would affect the generated speech or sound effects.

  3. The Expert-Pipeline directly averages independently generated waveforms without loudness normalization or other audio mixing techniques. This may lead to unbalanced loudness levels and unnatural interactions between different audio components, potentially affecting the final perceptual quality. Therefore, it is unclear how much of the reported improvement comes from joint generation rather than the limitations of the baseline mixing strategy.

  4. The proposed model underperforms several existing methods on AudioCaps, suggesting that its general text-to-audio generation capability may be limited compared with specialized models. In addition, the strong results on MusicCaps are mainly supported by automatic metrics such as FAD and CLAP, which may not adequately capture fine-grained musical characteristics, including melody, rhythm, and instrumentation. Although human evaluation is conducted on MECAT, a direct perceptual comparison with specialized music generation models on MusicCaps is still missing.

Questions for authors
  1. Can the model generate plausible background sounds from a high-level scene description, even when individual audio components are not explicitly specified? Has the model been trained or evaluated under such underspecified conditions?

  2. Could the authors provide a direct human evaluation against specialized music generation models on MusicCaps? Since FAD and CLAP may not be sufficiently sensitive to fine-grained musical characteristics, a perceptual evaluation focusing on music quality and text alignment would help support the reported improvements.

  3. Does the reported 1.5-second generation latency include text encoding, audio decoding, and the optional LLM-based prompt planning/refinement? What is the complete end-to-end latency when generating an audio scene from a natural-language description?

  4. Compared with WavJourney, which supports explicit temporal composition, how does the proposed model handle the timing and ordering of different sound events? This is particularly relevant given its lower multiFLAM scores in most mixed-audio categories.

  5. Could the authors evaluate the Expert-Pipeline using a more carefully designed mixing strategy, such as loudness normalization and ducking? It would be useful to understand whether loudness imbalance and unnatural transitions between audio components significantly affect the baseline performance.

5 comments

Thank you for recognizing the simplicity of our framework and the breadth of our evaluation. We add the following experiments and clarifications.

1. Semantic Information and Latent Capacity (W1)

We added semantic-only, acoustic-only, and full DashengTokenizer comparisons. All generation models were trained on ACAVCaps for 800K steps. We report reconstruction, single-domain generation, and MECAT SMA results below. These ablations show that increasing acoustic latent capacity improves some mixed-scene generation metrics. At the same latent capacity, adding semantic priors further improves all listed single-domain metrics and mixed-scene FAD, CLAP, and GLAP, demonstrating benefits beyond increased capacity.

The acoustic baseline uses the VAE released with UniFlow-Audio: 24 kHz mono input and 128D latents at 50 Hz. DashengTokenizer uses 1280D latents at 25 Hz.

Audio reconstruction

Latent representationSeedTTS-en PESQ ↑SeedTTS-en STOI ↑SeedTTS-zh PESQ ↑SeedTTS-zh STOI ↑MUSDB18-HQ Mel ↓MUSDB18-HQ STFT ↓AudioSet Mel ↓AudioSet STFT ↓
DashengTokenizer4.0870.85644.1080.86080.45721.8260.48061.964
DashengTokenizer (semantic only)2.2040.90842.2900.90281.16382.9131.13292.784
DashengTokenizer (acoustic only)4.1630.99054.1660.99080.20751.4880.20971.598
VAE3.8580.99204.0480.99000.23201.1290.32101.385

Single-domain generation

Latent representationAudioCaps FAD ↓MusicCaps FAD ↓LibriTTS WER (%) ↓LibriTTS UTMOSv2 ↑
DashengTokenizer3.2011.39111.043.078
DashengTokenizer (semantic only)3.2842.15127.902.949
DashengTokenizer (acoustic only)6.2625.01935.612.827
VAE4.3822.90433.401.488

MECAT SMA

Latent representationFAD ↓CLAP ↑GLAP ↑MultiFLAM ↑WER (%) ↓UTMOSv2 ↑
DashengTokenizer2.3650.45410.3260.70528.232.443
DashengTokenizer (semantic only)2.2780.4978.8030.58933.122.537
DashengTokenizer (acoustic only)3.7410.4356.8080.78628.002.633
VAE2.8100.4539.3820.51938.032.146

Role of greater acoustic capacity. Compared with the 128D VAE, the 1280D acoustic-only branch improves only LibriTTS UTMOSv2 among the listed single-domain metrics, but improves MultiFLAM, WER, and UTMOSv2 on MECAT SMA. FAD, CLAP, and GLAP do not improve. Thus, increasing acoustic capacity alone does not improve generation across all metrics.

Benefit of semantic priors at equal capacity. The full and acoustic-only DashengTokenizer representations have the same latent capacity. Although the acoustic-only branch is better on all eight reconstruction metrics, the full representation is better on every listed single-domain generation metric. On MECAT SMA, it also improves FAD, CLAP, and GLAP while retaining similar WER. These gains cannot be explained by greater capacity or better reconstruction, supporting the added value of semantic priors for conditional generation.

Capacity and semantic information are complementary. Expanding the acoustic latent space mainly improves component alignment (MultiFLAM), speech intelligibility (WER), and predicted speech quality (UTMOSv2) in mixed scenes. Adding semantic priors at equal capacity further improves distribution quality (FAD), text alignment (CLAP, GLAP), and single-domain generation (AudioCaps and MusicCaps FAD; LibriTTS WER and UTMOSv2). Our design therefore combines acoustic capacity with semantic information, rather than relying on either alone. We will add these experiments and analysis to the revision.

2. Selective Control (W2)

We added paired single-field edits to test whether the target changes as requested while other content is preserved. We used 50 prompts covering different themes, each with five control fields. Each pair changes one field and fixes all other fields and the initial noise. With three paired seeds, this yields 1,500 audio clips.

Metrics. Let A and B denote the original and edited conditions, and s denote CLAP cosine similarity:

  • Target response: Δtarget = [s(B audio, B description) − s(B audio, A description)] − [s(A audio, B description) − s(A audio, A description)]. Positive values indicate a shift toward the edited target.
  • Non-target change: Δpreserve = s(B audio, unchanged description) − s(A audio, unchanged description). Values near zero indicate little score change; negative values indicate reduced alignment.
  • Transcript column: ΔWER = WER(B) − WER(A).

Results. Cells report means and 95% bootstrap intervals. Target responses are bold.

Edited field ↓ / Evaluated field →Music (CLAP)SFX (CLAP)Acoustics (CLAP)Speech attributes (CLAP)Transcript (ΔWER, pp)
Music+10.85 [6.20, 15.11]−2.34 [−3.72, −0.94]+1.95 [0.14, 4.17]+0.63 [−0.67, 1.93]−0.10 [−3.08, 2.93]
SFX−0.99 [−4.16, 2.19]+8.92 [4.47, 13.57]−1.49 [−3.34, 0.32]−0.63 [−2.58, 1.42]−1.88 [−6.00, 1.88]
Acoustics−1.05 [−2.59, 0.50]−3.70 [−6.24, −1.32]+3.30 [1.71, 4.76]+0.43 [−1.33, 2.00]−0.35 [−3.97, 3.42]
Speech attributes−1.79 [−3.17, −0.54]+1.87 [0.23, 3.51]+0.02 [−1.12, 1.00]+1.40 [0.79, 2.08]−0.30 [−2.48, 1.64]
Transcript+0.87 [−0.95, 2.63]−2.39 [−5.45, 0.63]−0.73 [−2.46, 1.22]+0.29 [−1.88, 2.59]+4.86 [−6.05, 15.72]

Target CLAP responses are significantly positive for all four descriptive fields. Editing the transcript causes no significant change in any of the four non-target CLAP scores. Editing other fields changes mean WER by −1.88 to −0.10 percentage points, with no significant changes.

Of the 20 non-target responses, 15 are not significant, and their changes are smaller than target responses. These results support effective single-field control. We will add this experiment to the revision.

3. Improved Expert-Pipeline Mixing (W3, Q5)

We compared waveform averaging, loudness normalization, and normalization plus speech-driven ducking using fixed Expert-Pipeline stems. Mixing parameters were set on separate development samples.

All methods use the same stems, converted to 16 kHz mono, aligned at the start, and padded or trimmed to 10 seconds:

  • Waveform averaging: Sum the available stems sample by sample and divide by the number of stems.
  • Loudness normalization: Adjust each stem to −23 LUFS integrated loudness, with gain capped at +12 dB, then average as above.
  • Normalization + ducking: After normalization, use energy-based VAD on the original speech stem to detect speech. Attenuate music and sound effects toward −9 dB during speech and restore them afterward, with exponential smoothing (20 ms attack, 250 ms release). Average by the same stem count. Without speech, this reduces to loudness normalization.
System / Mixing methodFAD ↓FD(PANNs)↓CLAP ↑GLAP ↑WER (%) ↓UTMOSv2 ↑
Expert: waveform averaging6.23843.86623.098.6157.832.29
Expert: loudness normalization6.42543.64022.178.9941.532.37
Expert: normalization + ducking5.85340.49221.728.5228.042.58
Dasheng AudioGen2.73425.38823.619.0428.152.53

Compared to the best Expert-Pipeline (normalization + ducking), Dasheng AudioGen maintains highly competitive speech intelligibility (WER 28.15% vs. 28.04%; UTMOSv2 2.53 vs. 2.58), while significantly advancing the overall audio distribution quality (FAD: 2.734 vs. 5.853; FD: 25.388 vs. 40.492) and text alignment (CLAP: 23.61 vs. 21.72; GLAP: 9.04 vs. 8.52). This confirms that our single-pass joint generation model not only eliminates the need for intricate mixing strategies but also better balances intelligibility, acoustic realism, and semantic alignment.

4. AudioCaps Performance and Music Quality (W4, Q2)

We acknowledge that our model does not yet outperform some specialized methods on AudioCaps. Our goal is to balance single-domain ability with complex mixed-scene generation; the results reflect this trade-off in unified modeling. We are also conducting MusicCaps listening tests and aim to report the results before the rebuttal deadline.

5. Underspecified Input and Background Completion (Q1)

Dasheng AudioGen is trained with missing conditions: the global caption is retained, while each of the other five fields is independently dropped with probability 0.2.

On MECAT SMA, we compared all original fields with only the original <|caption|>, using the same initial noise for each pair.

InputFAD ↓FD(PANNs)↓OpenFLAM ↑CLAP ↑GLAP ↑UTMOSv2 ↑
All original fields2.99922.99567.1827.409.652.630
Only <|caption|>4.76626.44362.4130.6911.542.346

With only the original caption, the model retains good scene-text alignment: CLAP reaches 30.69 and GLAP 11.54, both higher than with all fields. Adding the full fields reduces FAD from 4.766 to 2.999 and FD from 26.443 to 22.995, while increasing OpenFLAM by 4.78 points and UTMOSv2 by 0.284. The model can thus handle caption-only input, while explicit component descriptions further improve component alignment, distribution quality, and predicted speech quality.

6. Generation Latency (Q3)

The 1.5-second latency includes text encoding, DiT sampling, audio decoding, and file writing, but excludes the LLM prompt refiner request. With the DeepSeek-V4-Flash online API as the prompt refiner and a single H200 GPU, the mean end-to-end time from a natural-language description to an audio scene is 35.46 seconds.

7. Event Timing and Order (Q4)

Dasheng AudioGen learns temporal relations implicitly from large-scale real mixed audio and uses natural-language conditions to guide event order. Words such as “first,” “then,” and “after” can describe sequences such as “a knock, followed by someone speaking.” Unlike WavJourney's explicit scheduling, our model jointly generates how events unfold without assigning individual start and end times.

multiFLAM measures alignment with component descriptions and takes the maximum response over time. A lower score therefore does not directly indicate weaker event-order control. We will clarify this metric's scope and leave explicit timestamps and event durations to future work.

Reviewer_Ppx4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dasheng AudioGen generates mixed audio scenes using structured captions and a flow-matching DiT operating in DashengTokenizer’s semantic-acoustic latent space. Caption fields describe speech, music, sound effects, and environmental context, which condition a shared model to generate the combined scene. Experiments show improved distributional quality and speech intelligibility over compositional baselines, supported by caption and representation ablations, public-data training, and human evaluation.

Critical strengths
  1. Table 3 shows improved distributional quality and lower WER than WavJourney across speech-containing mixtures. On SMA, FAD improves from 6.95 to 2.17 and WER from 34.06% to 28.98%. These simultaneous gains support the practical value of joint scene generation when speech must remain intelligible alongside music and sound effects. Human evaluation with 20 audio-domain professionals also favors Dasheng AudioGen over Expert-Pipeline, providing complementary evidence of perceptual quality.
  2. Appendix E provides a useful control by retaining identical textual information while removing field markers. Structured conditioning improves nine of ten metrics, including a reduction in LibriTTS WER from 55.28% to 17.14%. This supports the usefulness of explicit field markers beyond simply providing richer descriptions. The representation comparison uses the same DiT backbone for mixed-data training, and the public-subset experiment demonstrates effective training with approximately 10K hours of accessible data.
Critical weaknesses
  1. The contribution needs clearer positioning relative to structured generation in UALM[1] and Foley-Omni[2], and unified generation in Audio-Omni[3], and Fugatto[5], all of which also support speech, music, and sound generation. The paper should specify what its caption interface and joint mixed-scene modeling add beyond these systems. A comparison with an available unified model would strengthen this positioning where conditioning interfaces and supported outputs are comparable.
  2. The representation ablation supports the complete tokenizer choice but does not isolate why it helps. Appendix C compares 128-dimensional acoustic VAE latents with 1280-dimensional DashengTokenizer latents, changing both latent capacity and tokenizer pretraining. Keeping the DiT backbone fixed is useful, but improvements cannot be attributed specifically to semantic priors without separating these factors. This limits the mechanistic explanation instead of the reported empirical gains.
  3. The evidence for selective control is weaker than that for improved generation. The caption ablation changes the conditioning format globally. It does not test whether varying a field produces the requested attribute change while retaining unrelated content. This matters particularly for environmental acoustics. Matched examples varying one field would clarify how reliably the model follows these controls.
  4. Component coverage and acoustic coherence remain incompletely evaluated. The paper acknowledges the quality alignment trade-off, but WavJourney’s higher multiFLAM scores across speech containing mixtures, including 86.09 versus 71.40 on SMA, still warrant component-level analysis. Human evaluation excludes WavJourney and uses an Expert-Pipeline with simple waveform averaging, limiting perceptual evidence against a stronger compositional system. PAFI’s agreement with human ratings is measured across category-level system comparisons. It supports system ranking without directly validating acoustic properties.

[1] Ualm: Unified audio language model for understanding, generation and reasoning. ICLR 2026

[2] Foley-Omni: A Unified Multimodal Generation Model from Task-Level Audio Synthesis to Complete Video Soundtrack Generation

[3] Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing. SIGGRAPH 2026

[4] Fugatto 1: Foundational Generative Audio Transformer Opus 1. ICLR 2025

Questions for authors
  1. Can matched examples that vary only a music, sound-effect, or environment field demonstrate the intended change while preserving the other requested content?
  2. Can component-level analysis explain the multiFLAM gap relative to WavJourney, distinguishing incomplete generation from acoustic masking or sensitivity of the similarity metric?
  3. Which checkpoint and latent frame rate were used for the acoustic VAE baseline? What evidence, such as a capacity matched control, separates semantic-prior benefits from increased latent capacity?
  4. Could WavJourney be included in a targeted listening comparison covering component completeness and scene coherence, using the existing evaluation examples?
  5. What evidence attributes the LibriTTS WER gap mainly to truncation after references are filtered to under ten seconds? How often does generated speech reach the output boundary?
6 comments

Thank you for recognizing the benefits of joint scene generation and the information-matched caption ablation. We address your main concerns below.

1. Contribution and Unified-Model Comparison (W1)

The key difference between our work and existing unified audio models is that we jointly generate intelligible speech, music, and sound effects from text in a single end-to-end pass, rather than supporting these tasks separately. Under this definition of mixed-scene generation, only Foley-Omni and Dasheng AudioGen are comparable. We will make this distinction explicit in the revision.

ModelIntelligible SpeechTTATTMMixed AudGenMixed Audio EvalOpen Source
UALM✗✓✓✗✗✗
Foley-Omni✓✓✓✓✓✓
Audio-Omni✓✓✓✗✗✓
Fugatto 1✓✓✓✗✗✗
Dasheng AudioGen (Ours)✓✓✓✓✓✓

UALM does not generate intelligible speech with controlled text content. Audio-Omni supports TTS, TTA, and TTM separately, but not jointly. Fugatto 1 combines tasks through ComposableART, but its public demos show only pairs such as speech + sound effects or music + sound effects. Neither its paper nor demos show all three together, and its weights are not public.

Foley-Omni focuses on video-and-text-conditioned audio and audiovisual synchrony, whereas we focus on text-driven mixed scenes. We compared both models on MECAT using Foley-Omni's public text-only interface. Dasheng AudioGen performs better on all five metrics, supporting our design for text-conditioned mixed-scene generation.

DatasetModelOpenFLAM ↑CLAP ↑GLAP ↑WER (%) ↓UTMOSv2 ↑
MECATDasheng AudioGen78.4023.619.0428.152.53
MECATFoley-Omni73.2720.676.5730.672.06

2. Representation Choice and VAE Configuration (W2, Q3)

We added semantic-only, acoustic-only, and full DashengTokenizer comparisons. All generation models were trained on ACAVCaps for 800K steps. We report reconstruction, single-domain generation, and MECAT SMA (speech + music + sound effects) results below. These ablations show that increasing acoustic latent capacity improves some mixed-scene generation metrics. At the same latent capacity, adding semantic priors further improves all listed single-domain metrics and mixed-scene FAD, CLAP, and GLAP, demonstrating benefits beyond increased capacity.

The acoustic baseline uses the VAE released with UniFlow-Audio: 24 kHz mono input and 128D latents at 50 Hz. DashengTokenizer uses 1280D latents at 25 Hz.

Audio reconstruction

Latent representationSeedTTS-en PESQ ↑SeedTTS-en STOI ↑SeedTTS-zh PESQ ↑SeedTTS-zh STOI ↑MUSDB18-HQ Mel ↓MUSDB18-HQ STFT ↓AudioSet Mel ↓AudioSet STFT ↓
DashengTokenizer4.0870.85644.1080.86080.45721.8260.48061.964
DashengTokenizer (semantic only)2.2040.90842.2900.90281.16382.9131.13292.784
DashengTokenizer (acoustic only)4.1630.99054.1660.99080.20751.4880.20971.598
VAE3.8580.99204.0480.99000.23201.1290.32101.385

Single-domain generation

Latent representationAudioCaps FAD ↓MusicCaps FAD ↓LibriTTS WER (%) ↓LibriTTS UTMOSv2 ↑
DashengTokenizer3.2011.39111.043.078
DashengTokenizer (semantic only)3.2842.15127.902.949
DashengTokenizer (acoustic only)6.2625.01935.612.827
VAE4.3822.90433.401.488

MECAT SMA

Latent representationFAD ↓CLAP ↑GLAP ↑MultiFLAM ↑WER (%) ↓UTMOSv2 ↑
DashengTokenizer2.3650.45410.3260.70528.232.443
DashengTokenizer (semantic only)2.2780.4978.8030.58933.122.537
DashengTokenizer (acoustic only)3.7410.4356.8080.78628.002.633
VAE2.8100.4539.3820.51938.032.146

Role of greater acoustic capacity. Compared with the 128D VAE, the 1280D acoustic-only branch improves only LibriTTS UTMOSv2 among the listed single-domain metrics, but improves MultiFLAM, WER, and UTMOSv2 on MECAT SMA. FAD, CLAP, and GLAP do not improve. Thus, increasing acoustic capacity alone does not improve generation across all metrics.

Benefit of semantic priors at equal capacity. The full and acoustic-only DashengTokenizer representations have the same latent capacity. Although the acoustic-only branch is better on all eight reconstruction metrics, the full representation is better on every listed single-domain generation metric. On MECAT SMA, it also improves FAD, CLAP, and GLAP while retaining similar WER. These gains cannot be explained by greater capacity or better reconstruction, supporting the added value of semantic priors for conditional generation.

Capacity and semantic information are complementary. Expanding the acoustic latent space mainly improves component alignment (MultiFLAM), speech intelligibility (WER), and predicted speech quality (UTMOSv2) in mixed scenes. Adding semantic priors at equal capacity further improves distribution quality (FAD), text alignment (CLAP, GLAP), and single-domain generation (AudioCaps and MusicCaps FAD; LibriTTS WER and UTMOSv2). Our design therefore combines acoustic capacity with semantic information, rather than relying on either alone. We will add these experiments and analysis to the revision.

3. Single-Field Control (W3, Q1)

We added paired single-field edits to test whether the target changes as requested while other content is preserved. We used 50 prompts covering different themes, each with five control fields. Each pair changes one field and fixes all other fields and the initial noise. With three paired seeds, this yields 1,500 audio clips.

Metrics. Let A and B denote the original and edited conditions, and s denote CLAP cosine similarity:

  • Target response: Δtarget = [s(B audio, B description) − s(B audio, A description)] − [s(A audio, B description) − s(A audio, A description)]. Positive values indicate a shift toward the edited target.
  • Non-target change: Δpreserve = s(B audio, unchanged description) − s(A audio, unchanged description). Values near zero indicate little score change; negative values indicate reduced alignment.
  • Transcript column: ΔWER = WER(B) − WER(A).

Results. Cells report means and 95% bootstrap intervals. Target responses are bold.

Edited field ↓ / Evaluated field →Music (CLAP)SFX (CLAP)Acoustics (CLAP)Speech attributes (CLAP)Transcript (ΔWER, pp)
Music+10.85 [6.20, 15.11]−2.34 [−3.72, −0.94]+1.95 [0.14, 4.17]+0.63 [−0.67, 1.93]−0.10 [−3.08, 2.93]
SFX−0.99 [−4.16, 2.19]+8.92 [4.47, 13.57]−1.49 [−3.34, 0.32]−0.63 [−2.58, 1.42]−1.88 [−6.00, 1.88]
Acoustics−1.05 [−2.59, 0.50]−3.70 [−6.24, −1.32]+3.30 [1.71, 4.76]+0.43 [−1.33, 2.00]−0.35 [−3.97, 3.42]
Speech attributes−1.79 [−3.17, −0.54]+1.87 [0.23, 3.51]+0.02 [−1.12, 1.00]+1.40 [0.79, 2.08]−0.30 [−2.48, 1.64]
Transcript+0.87 [−0.95, 2.63]−2.39 [−5.45, 0.63]−0.73 [−2.46, 1.22]+0.29 [−1.88, 2.59]+4.86 [−6.05, 15.72]

Target CLAP responses are significantly positive for all four descriptive fields. Editing the transcript causes no significant change in any of the four non-target CLAP scores. Editing other fields changes mean WER by −1.88 to −0.10 percentage points, with no significant changes.

Of the 20 non-target responses, 15 are not significant, and their changes are smaller than target responses. These results support effective single-field control. We will add this experiment to the revision.

Comments on the paper

2 comments
Area_Chair_zvwNOpenReview ↗

Discussion phrase will end soon

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

AuthorsOpenReview ↗

Summary of Rebuttal Progress

Dear Area Chair,

Thank you for coordinating the review process and encouraging reviewer engagement and discussion! We briefly summarize the rebuttal progress:

  • The main technical concerns have been addressed. We added capacity-matched representation ablations, single-field control experiments, stronger mixing baselines, and LibriTTS truncation analysis to address questions about the sources of improvement, controllability, and evaluation fairness. Additional listening tests provide direct evidence of mixed-scene perceptual quality and non-target preservation. Reviewer JaMy confirmed that the main concerns were resolved and raised the score from 3 to 4. Reviewer Ppx4 stated that the new evidence addressed most concerns and maintained their acceptance recommendation.
  • The remaining disagreement mainly concerns the contribution’s scope. Reviewer wn15 acknowledged the thorough experiments and the soundness of our evaluation but remains concerned about novelty. We clarified that DashengTokenizer and ACAVCaps provide the representation and data foundations, while our work studies conditional generation from real mixed audio through conditioning design, representation comparisons, and dedicated evaluation, supported by publicly available models and code.

We will incorporate the additional experiments and discussions into the revised manuscript and clarify the contribution boundaries. We respectfully ask you to consider the paper’s technical soundness, its contribution to mixed-scene generation, and the additional evidence in your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.