acceptodds
Under review as a conference paper at ICLR 2027

Building Audio Scenes with Voice Design and Cloning

Abstract

Building audio scenes requires jointly generating multi-speaker speech, ambience, music, and sound effects. This task requires consistent and expressive speech coordinated with the surrounding audio. Speech can be generated through voice design from a scene description or through voice cloning from reference audio. However, the scarcity of high-quality annotated data and the difficulty of supporting both tasks while jointly generating multiple audio modalities further complicate audio scene generation. Therefore, we first propose VoxCaps, which cleans media data, expands it with targeted synthetic data, and uses a persona library to annotate diverse multi-level scene descriptions. Then, we introduce VoxVAE, which pairs a local anti-aliased encoder with a high-capacity decoder to provide a shared continuous representation for multiple audio modalities. Next, we propose VoxDirector, an expressive multi-speaker speech and surrounding audio generation model that supports both voice design and cloning. VoxDirector adopts reward-conditioned quality control to improve generation quality, and a Dual-Router MoE tailored to the two tasks and multiple audio modalities. Experimental results show that VoxDirector outperforms the baselines on key metrics for both voice design and cloning, while supporting the generation of complex audio scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Speech, Text-to-Audio, Controllable Speech Synthesis, Zero-Shot TTS, Dialogue Synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TNCPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper proposes VoxDirector, which is a flow-matching DiT that generates multi-speaker speech with diverse acoustic backgrounds. The system supports voice design from a structured description and voice cloning from reference audio. Its components include VoxCaps (a media-derived data pipeline with a persona library and human Best-Worst scaling), VoxVAE (25-Hz continuous 48-kHz latents), quality-metadata conditioning, a dual-router MoE, and a four-stage curriculum. Evaluation covers a wide range of tasks, from reconstruction, cloning, voice design and full scenes.

Critical strengths
  • The work is good as a system design: one model for design, cloning, dialogue and non-speech audio. The Environment / Speakers / Content format cleanly separates stable traits from local delivery.
  • Detailed appendices and well-motivated targeted synthesis are plus.
Critical weaknesses
  • The contribution is a large system with limited isolated evidence for each component. The main generative ablations (Table 5) are all on one in-house benchmark scored by one LLM judge. It is hard to tell which design choices matter and why. For ICLR, I would expect clearer evidence of what is learned further beyond “this system works well.”
  • Claims exceed the results: (1) VoxDirector does not win baselines on InstructTTSEval RP and DSD, despite the paper’s role-play motivation. (2) VoxCPM2 is missing from VoxBench-Instrut though highlighted ahead. (3) The Table 3 baseline numbers are copied from another report, which is okay unless the same judge are used, which would need double checks.(4) Content error is higher than that of TTS-focused systems, which deserves additional explanation.
  • The most important scene evaluation is thin and not clear. Basically, voxbench-scene has only two weak baselines with LLM judge, which is also in-house only without other chance to fully understand the performance gain.

minor concerns:

  • Apart from VoxDirector itself, the named parts are largely combinations of existing tools:
    • VoxCaps connects off-the-shelf models: UVR, 3D-Speaker, Fun-ASR, Qwen3-ForcedAligner, Qwen3.5-Omni, SQUIM, and CosyVoice 3.
    • VoxVAE combines SAME-style objectives, EnCodec-style augmentation, and standard discriminators.
    • VoxBench-Instruct and VoxBench-Scene are small in-house test sets.

None of these is evaluated or released as an independent artifact, and VoxCaps is not compared against simpler annotation pipelines. Naming each one implies a standalone contribution the paper does not establish. They should be described as components of the system.

  • The model is conditioned on SQUIM-derived quality and then evaluated with SQUIM-PESQ. This could introduce a potential issue that the fidelity evaluation may be gamed.
  • Expressiveness and all ablations are scored by LLM judges. Judge–human agreement is a Spearman correlation of 0.64–0.72, measured on only 100 clips. No confidence intervals, significance tests, or seeds are reported, and several ablation gaps are small. All above indicate a over-reliance on LLM judge. A bit more information with human checks would be helpful here.
Questions for authors

Covered in the weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer TNCP (1/3)

Thank you for your comments. In this response, we add human listening evaluations for VoxCPM2 and a five-system scene evaluation that includes ScenA and UNISON. We address your concerns below.

1. Method contributions, component evaluation, and what the model learns

Thank you for your interest in the method's novelty and the effectiveness of its components.

We study complete audio-scene generation at four levels (data supervision, acoustic representation, the generative model, and evaluation benchmarks) through VoxCaps, VoxVAE, VoxDirector, and VoxBench, respectively.

Compared with annotating an entire recording with a single overall description, VoxCaps labels persistent ambience, stable character attributes, and utterance-level emotions and sound effects separately, using a character-archetype style matrix to organize voice and performance cues. The matrix covers character types in advertising, animation, and short drama, with descriptions grounded in the actual audio. The four-stage processing code, hierarchical prompts, and style matrix are available in the VoxCaps repository (Section 3.3; Appendix A.3).

Building on this structured supervision, VoxDirector introduces two core designs for unified scene generation:

  • Dual-granularity task–acoustic routing. The Task Router selects a voice-design or cloning expert at the sample level. The Audio Router dynamically selects acoustic experts according to frame-level acoustic content and the denoising stage. Dynamic Top-P selection and null routes allow each frame to adjust the number of active experts; the task expert remains active when the acoustic branch is skipped. Routing results show that low-pitched ambience is assigned mainly to E1 (75.20%), higher-frequency instruments to E2 (74.28%), male speech to E0 (62.77%), and female speech to E3 (64.87%). In Table 5, removing the Audio Router lowers instruction accuracy, acoustic quality, and expressiveness from 3.61 / 4.33 / 3.89 to 3.36 / 4.03 / 3.59.
  • Implicit reinforcement learning through outcome conditioning. Recording quality and expressiveness do not always coincide in real-world scene data. We convert offline quality evaluations into quality descriptions and a global quality level, supplied to the flow-matching model through cross-attention and AdaLN-Zero, and specify high-quality conditions at inference. Recordings of moderate quality but rich expressiveness can therefore still contribute to style learning, while the desired output quality is controlled through conditioning. In Table 5, removing quality conditioning lowers acoustic quality from 4.33 to 3.64 (Section 4.3; Appendix C.2).

VoxVAE has objective evaluations and ablations independent of the downstream scene generator. Table 1 evaluates speech, general-audio, and music reconstruction on VCTK, FSD50K, and MUSDB18-HQ, respectively; Appendix E.2 ablates its training objectives. Removing the acoustic readout lowers reference-based PESQ from 4.168 to 3.768 and increases MCD from 0.964 to 1.332.

VoxBench contributes defined evaluation tasks for role-play and complete audio-scene generation. VoxBench-Instruct covers advertising, short-drama, and general scenarios, evaluating expressiveness, prosodic naturalness, audio fullness, and scene appropriateness. VoxBench-Scene organizes cases by non-speech content, speaker count, local effects, and reference-voice conditions, comparing instruction following, acoustic quality, and expressiveness within scenes. The two benchmarks assess character performance and multi-source scene generation, respectively; their composition and scoring protocols are described in Appendices D.3 and D.4.

AuthorsOpenReview ↗

Response to Reviewer TNCP (2/3)

2. VoxCPM2 comparison and human evaluation of role-play

Thank you for your attention to role-play evaluation and for suggesting the VoxCPM2 comparison.

Some role instructions in InstructTTSEval allow multiple valid interpretations, and its coverage of emotions and character types is uneven. For example, “explain a problem in a teacher's voice” can reasonably be performed with either warmth and patience or seriousness and firmness. Evaluators may interpret such instructions differently, while a simple true/false judgment cannot fully capture subtle differences in delivery. Human listening evaluation is therefore also needed to assess role-play expressiveness.

The benchmark's instructions also do not cover joint control of multi-speaker dialogue and environmental sound effects. VoxDirector is highly competitive on InstructTTSEval, achieving the highest Chinese and English APS scores among the systems in Table 3.

Table 3 follows InstructTTSEval's Gemini 2.5 Pro judging protocol, with APS, DSD, and RP calculated according to the benchmark definitions. The evaluation method is described in Appendix D.3.

VoxBench-Instruct includes advertising, short-drama, and general scenarios. It evaluates role-play through expressiveness, prosodic naturalness, audio fullness, and scene appropriateness. The test set contains 180 cases, with five listeners rating each evaluated output. The complete results after adding VoxCPM2 are as follows:

ModelMean MOSExpressivenessProsodic naturalnessAudio fullnessScene appropriateness
MiMo-Audio3.213.063.273.572.94
MOSS-VoiceGenerator3.603.423.624.063.31
VoxCPM23.953.833.934.303.73
Qwen3-TTS-VD4.093.964.094.453.88
VoxDirector4.224.124.204.474.10

VoxDirector has the highest overall human mean and the highest scores on all four individual dimensions, clearly outperforming VoxCPM2. On VoxBench-Scene, which covers multi-speaker speech, ambience, and local sound effects, VoxDirector also achieves the highest reported means on all three human-rated dimensions in all five categories.

3. Content accuracy and multi-task generation

Thank you for your attention to content accuracy. However, characterizing VoxDirector's content accuracy as lower than that of specialized TTS systems is not accurate.

For example, on SwanBench-Speech (Table 2), VoxDirector's single-speaker Error is 0.086, below CosyVoice-2 (0.106) and ZipVoice (0.213); its multi-speaker Error is 0.120, below FireRedTTS-2 (0.148) and VibeVoice (0.204). On Seed-TTS-Eval (Appendix E.3), its Chinese CER is 1.21%, below F5-TTS (1.53%); its English WER is 2.34%, below MaskGCT (2.62%) and CosyVoice 2 (3.09%).

Beyond content accuracy, VoxDirector supports both instruction-driven voice design (Instruct) and zero-shot voice cloning (Zero-shot), covering single-speaker and multi-speaker speech as well as the joint generation of speech, ambience, and sound effects. We therefore cover a much broader range of generation tasks and scenarios. Regarding expressiveness, timbre consistency, expressive richness, and expressive hierarchy all achieve the highest reported values in their respective groups in the voice-cloning evaluation (Table 2).

5. SQUIM in training and independent quality evaluation

Thank you for your attention to the independence of quality evaluation.

During training, SQUIM provides quality estimates in offline data preprocessing, which are used to construct discrete quality descriptions and a global level. Implicit reinforcement learning follows an outcome-conditioning mechanism, with the generator learning audio distributions under the corresponding conditions. The training objective remains flow matching; we neither backpropagate SQUIM gradients through generated audio nor maximize SQUIM rewards through online rollouts.

During evaluation, SQUIM-PESQ corresponds only to the Fidelity column in Table 2. PESQ and STOI in Table 1 and Appendix E.2 use reference-based calculations. Timbre similarity, SRMR, ASR, SpeechJudge, expressiveness evaluation, and human listening evaluations do not rely on SQUIM.

Across the five scene categories evaluated by both VoxDirector and the cascade, VoxDirector's mean human-rated quality ranges from 4.42 to 4.52, compared with 3.95 to 4.13 for the cascade, an improvement of 0.29 to 0.54 points in each category.

AuthorsOpenReview ↗

Response to Reviewer TNCP (3/3)

4. Human evaluation and scene comparisons

Thank you for suggesting additional human listening evaluations and scene baselines.

We added ScenA and UNISON as unified generation baselines alongside Dasheng AudioGen and the Qwen3-TTS 1.7B + TangoFlux cascade. VoxBench-Scene contains 500 cases in five categories of 100 each. The no-dialogue category has no references; each of the other four categories contains 50 description-only cases and 50 reference-conditioned cases. Five listeners rate each evaluated output, and results are reported as means ± 95% confidence intervals:

SceneModelnHuman instruction accuracyHuman qualityHuman expressiveness
No dialogue + local effectsQwen3-TTS + TangoFlux1003.50±0.193.99±0.083.61±0.14
Dasheng AudioGen1003.06±0.123.42±0.203.36±0.07
ScenA1003.37±0.062.92±0.153.63±0.11
UNISON1003.01±0.132.94±0.092.41±0.18
VoxDirector1003.72±0.174.51±0.123.99±0.05
Single speaker, no local effectsQwen3-TTS + TangoFlux1003.34±0.084.10±0.163.56±0.20
Dasheng AudioGen502.51±0.143.07±0.113.13±0.06
ScenA1003.21±0.052.83±0.173.24±0.13
UNISON1002.18±0.102.11±0.142.05±0.19
VoxDirector1003.85±0.164.46±0.073.98±0.12
Single speaker + local effectsQwen3-TTS + TangoFlux1003.22±0.114.13±0.203.51±0.08
Dasheng AudioGen502.80±0.053.39±0.133.24±0.17
ScenA1003.39±0.182.76±0.093.18±0.12
UNISON1002.04±0.112.39±0.062.20±0.15
VoxDirector1003.65±0.074.42±0.143.94±0.19
Multiple speakers, no local effectsQwen3-TTS + TangoFlux1003.37±0.133.95±0.183.76±0.10
Dasheng AudioGen502.15±0.093.01±0.122.93±0.20
ScenA963.35±0.142.98±0.052.59±0.11
VoxDirector1003.70±0.184.49±0.134.08±0.06
Multiple speakers + local effectsQwen3-TTS + TangoFlux1003.32±0.104.07±0.193.65±0.14
Dasheng AudioGen502.62±0.153.44±0.113.33±0.06
ScenA943.49±0.122.91±0.172.66±0.09
VoxDirector1003.67±0.164.52±0.054.10±0.13

n is the actual number of evaluated cases. VoxDirector and the cascade cover all cases; Dasheng evaluates cases without references, UNISON is included only in no-dialogue and single-speaker scenes, and ScenA binds at most three reference speakers.

VoxDirector has the highest reported mean on all three human-rated dimensions in all five categories. On the same 100 multi-speaker cases with local effects evaluated by both systems, it exceeds the cascade by 0.35, 0.45, and 0.45 points in instruction accuracy, acoustic quality, and expressiveness, respectively, with non-overlapping corresponding 95% confidence intervals. In a separate sound-field consistency check on 100 applicable cases, VoxDirector scores 97% and the cascade 36%.

The original Table 4 already contains direct human listening evaluations of expressiveness and three other dimensions, while the VoxVAE ablations in Appendix E.2 use objective reconstruction metrics. Thus, the statement that expressiveness and all ablations are evaluated by an LLM does not match the experimental setup in the paper. The added VoxCPM2 and scene listening evaluations further extend the cross-system comparisons. The paper also reports results on external benchmarks including InstructTTSEval and Seed-TTS-Eval.

6. Data authorization, consent for voice use, and safeguards

Thank you for your attention to source-audio licensing and authorization for voice use.

The source audio and voice uses involved in this study have the relevant authorization and consent. Materials are used within the scope of signed agreements. Public corpora follow their release licenses and original-subcorpus terms. For example, the VoxBox repository uses CC BY-NC-SA 4.0, with subcorpora used under their respective terms.

Custom test reference audio is collected independently and is not added to the training set; participants in subjective listening evaluations provide informed consent. The public VoxCaps repository provides data-processing code, hierarchical prompts, and annotation specifications. The use and redistribution of source audio remain subject to the applicable licenses and agreements.

The Ethics Statement explicitly discusses the risks of voice impersonation and misleading synthetic content and advocates obtaining permission for voice use, disclosing synthetic content, and protecting privacy and intellectual property.

Reviewer_mtbyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VoxDirector, a unified model for multi-speaker speech and audio scene generation, supporting both voice design and voice cloning. The authors introduce VoxCaps, a large-scale dataset with structured scene descriptions, VoxVAE for a shared continuous audio representation, and a flow-based Transformer with Dual-Router MoE. Experiments show promising results in voice cloning, voice design, and audio scene generation.

Critical strengths
  • A useful unified framework. The paper brings together voice design, voice cloning, multi-speaker dialogue, and surrounding audio generation in one model. This is practically useful and reduces the need for separate generation pipelines.

  • Solid system-level effort. The large-scale data pipeline and the design of VoxVAE and Dual-Router MoE are interesting. The experiments cover multiple tasks, and the ablations provide some evidence for the effectiveness of the proposed components.

Critical weaknesses
  • Limited novelty and insufficient validation of general audio generation. The overall framework largely follows existing latent flow-based audio generation methods. While the Dual-Router MoE is interesting, it is unclear whether it learns meaningful specialization across different audio types. More importantly, most evaluations focus on speech, while the results on complete audio scenes are aggregated. It is difficult to tell whether the improvements come mainly from stronger speech generation or better modeling of non-speech sounds and their interactions.

  • Insufficient evidence for data quality and evaluation independence. VoxCaps relies heavily on automatically generated annotations, while the automatic filtering mainly focuses on speech quality and annotation formatting. The more difficult aspects, such as sound event completeness and temporal consistency, appear to rely on human verification, but its coverage is unclear. In addition, the paper does not clearly explain how VoxBench-Scene is separated from the training data, making it difficult to assess generalization.

Questions for authors
  1. Could the authors report separate results on the non-speech-only, speech-with-sound-effects, and other mixed-audio subsets of VoxBench-Scene? This would help clarify whether the model improves general audio generation beyond speech synthesis.

  2. How is the semantic and temporal consistency between audio and scene descriptions verified at scale? What fraction of VoxCaps undergoes human verification, and are there any estimates of annotation error rates?

  3. How is VoxBench-Scene constructed, and what measures are taken to avoid overlap with VoxCaps in source recordings, speakers, or scene descriptions?

  4. Is there any evidence that the Audio Router learns meaningful specialization across different audio types, rather than simply benefiting from additional model capacity?

8 comments
AuthorsOpenReview ↗

Response to Reviewer mtby (1/2)

Thank you for your careful review and suggestions.

1. Non-speech and mixed-scene results

Thank you for suggesting separate results for non-speech, speech with effects, and other mixed scenes.

VoxBench-Scene contains 500 cases, divided into five categories of 100 according to the presence of dialogue, the number of speakers, and the presence of local effects. Five listeners rate each evaluated output. The table reports actual sample counts, mean human scores, and 95% confidence intervals for each system and category. Dasheng is evaluated on cases without reference audio; UNISON is included only in non-speech and single-speaker scenes; ScenA supports up to three reference speakers.

SceneModelnHuman instruction accuracyHuman qualityHuman expressiveness
No dialogue + local effectsQwen3-TTS + TangoFlux1003.50±0.193.99±0.083.61±0.14
Dasheng AudioGen1003.06±0.123.42±0.203.36±0.07
ScenA1003.37±0.062.92±0.153.63±0.11
UNISON1003.01±0.132.94±0.092.41±0.18
VoxDirector1003.72±0.174.51±0.123.99±0.05
Single speaker, no local effectsQwen3-TTS + TangoFlux1003.34±0.084.10±0.163.56±0.20
Dasheng AudioGen502.51±0.143.07±0.113.13±0.06
ScenA1003.21±0.052.83±0.173.24±0.13
UNISON1002.18±0.102.11±0.142.05±0.19
VoxDirector1003.85±0.164.46±0.073.98±0.12
Single speaker + local effectsQwen3-TTS + TangoFlux1003.22±0.114.13±0.203.51±0.08
Dasheng AudioGen502.80±0.053.39±0.133.24±0.17
ScenA1003.39±0.182.76±0.093.18±0.12
UNISON1002.04±0.112.39±0.062.20±0.15
VoxDirector1003.65±0.074.42±0.143.94±0.19
Multiple speakers, no local effectsQwen3-TTS + TangoFlux1003.37±0.133.95±0.183.76±0.10
Dasheng AudioGen502.15±0.093.01±0.122.93±0.20
ScenA963.35±0.142.98±0.052.59±0.11
VoxDirector1003.70±0.184.49±0.134.08±0.06
Multiple speakers + local effectsQwen3-TTS + TangoFlux1003.32±0.104.07±0.193.65±0.14
Dasheng AudioGen502.62±0.153.44±0.113.33±0.06
ScenA943.49±0.122.91±0.172.66±0.09
VoxDirector1003.67±0.164.52±0.054.10±0.13

All five systems evaluate the same 100 non-speech cases without dialogue and with local sound effects. VoxDirector achieves the highest mean scores for instruction accuracy (3.72), acoustic quality (4.51), and expressiveness (3.99), directly demonstrating its non-speech generation capability.

On the same 100 multi-speaker cases with local sound effects evaluated by both systems, VoxDirector exceeds the Qwen3-TTS + TangoFlux cascade by 0.35, 0.45, and 0.45 points on the three dimensions, with non-overlapping corresponding 95% confidence intervals. We also checked cross-source sound-field consistency on 100 applicable cases: whether speech, ambience, and effects sound as though they occupy the same acoustic space. VoxDirector scores 97%, compared with 36% for the cascade.

2. Large-scale annotation verification and error estimates

Thank you for asking about annotation quality verification.

After automatic annotation, we use Gemini to perform a second check against the original mixed audio. This checks semantic consistency between the description and the audio, including omitted sound events, incorrect speaker attribution, and whether the event order matches.

We also manually reviewed and calibrated approximately 570,000 audio clips, covering expressiveness, acoustic quality, spoken-content accuracy, and attributes including gender, age, and emotion. These human calibrations inform subsequent annotation quality control and expressive-data selection.

Human spot checks found a temporal-order error rate of approximately 3%.

AuthorsOpenReview ↗

Response to Reviewer mtby (2/2)

3. Test construction and separation from training

Thank you for asking us to explain the separation between VoxBench-Scene and the training data.

VoxBench-Scene is constructed independently of VoxCaps. Its scene instructions are newly written for evaluation, and reference recordings and voices are independently collected from external sources, with test speakers absent from the training data. These test instructions and reference materials were not used for training. The other evaluations use external public benchmarks or independently written test instructions; reference voices for our custom zero-shot evaluations are also collected separately from outside the training set. The instruction and reference-voice distributions in our custom evaluations are entirely different from those in training.

4. Evidence of Audio Router specialization

Thank you for asking whether the Audio Router learns meaningful acoustic specialization.

In the dual-router architecture, the Task Router selects a fixed expert for each sample according to the generation task, voice design or cloning. The Audio Router routes frame by frame according to acoustic features. To observe expert usage directly, we keep one checkpoint fixed, provide four types of audio input, and measure the allocation across four acoustic experts, averaged over multiple layers:

InputE0E1E2E3
Low-pitched ambience1.18%75.20%19.14%4.48%
Higher-frequency instruments1.05%18.71%74.28%5.96%
Single male voice62.77%11.03%6.34%19.86%
Single female voice22.92%8.27%3.94%64.87%

With model parameters fixed, the four input types produce different expert allocations: low-pitched ambience is routed mainly to E1 (75.20%), higher-frequency instruments to E2 (74.28%), male speech to E0 (62.77%), and female speech to E3 (64.87%). These results show that the Audio Router responds selectively to different acoustic inputs.

Table 5 separately reports the ablation that removes the Audio Router: scene instruction accuracy, acoustic quality, and expressiveness decrease from 3.61 / 4.33 / 3.89 to 3.36 / 4.03 / 3.59.

5. Methodological novelty

Thank you for suggesting that we clarify the novelty of the method.

Dialogue, ambience, and local sound effects in a complete audio scene share a timeline but change at different scales: character identity must remain stable across turns, expressive delivery and local effects change with the narrative, and the background may span the scene. VoxDirector jointly generates these sounds in a single latent sequence and supports both voice design from text and cloning from reference audio (Section 1). Its core designs are as follows:

  1. Style matrix and hierarchical acoustic conditioning. The VoxCaps style matrix maps character archetypes to audible voice and performance cues, explicitly distinguishing stable character attributes across turns from local delivery states within an utterance. During generation, transcripts are aligned to the latent timeline, while global and character-level descriptions enter the denoising network through cross-attention, allowing scene-spanning acoustic conditions and local delivery to participate jointly in generation (Sections 3.3 and 4.3; Appendix A.3).

  2. Dual-granularity task–acoustic routing. The Task Router selects a voice-design or cloning expert for the entire sample. The Audio Router selects acoustic experts frame by frame, with its routing budget adjusted over denoising stages. Dynamic Top-P selection and null paths allow different frames to use different numbers of acoustic experts; the acoustic branch can be skipped while the task expert remains active. The two routes thus handle sample-wide task conditions and frame-varying sounds separately (Section 4.4; Appendix C.4).

  3. Implicit reinforcement learning through outcome conditioning. We adopt an outcome-conditioned approach to implicit reinforcement learning, converting offline audio-quality evaluations into quality descriptions and a global quality level supplied through cross-attention and AdaLN-Zero, respectively. The model learns audio distributions under different quality conditions through flow-matching training. At inference, specifying high-quality conditions steers generation toward high-quality outputs. Recordings of moderate quality can therefore still contribute expressive styles, while the desired audio quality is explicitly specified as a generation condition (Section 4.3; Appendix C.2; Emmons et al., 2022).

Thank you for taking the time to review our paper. As the discussion period comes to a close, we’d appreciate hearing whether our responses addressed your concerns. If anything remains unclear or you have any further questions, we’d be happy to discuss them here.

Reviewer_jYTHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VoxDirector, a unified framework for generating expressive multi-speaker speech together with surrounding audio, including ambience, music, and sound effects. The framework supports both voice design from natural-language descriptions and voice cloning from reference audio.

The paper makes three main contributions. First, VoxCaps is introduced as a large-scale data pipeline that produces structured audio-scene annotations, separating environment descriptions, speaker characteristics, and local content. Second, VoxVAE provides a compact continuous representation shared across different audio domains. Third, VoxDirector employs a flow-based Transformer with a Dual-Router MoE architecture, reward-conditioned quality control, and curriculum learning.

Experimental results demonstrate competitive performance in voice cloning, voice design, and audio reconstruction. The proposed model also outperforms the evaluated baselines on the newly introduced VoxBench-Scene benchmark.

Critical strengths
  1. Meaningful and practically useful task. Jointly generating multi-speaker speech, ambience, music, and sound effects is valuable for applications such as animation, filmmaking, and content creation. Supporting both voice design and cloning within a unified framework is particularly appealing.
  2. Comprehensive system design. The paper presents a reasonably complete pipeline covering data collection, structured annotation, audio representation learning, model architecture, and multi-stage training. The large-scale VoxCaps dataset and its hierarchical descriptions are potentially valuable contributions.
  3. Competitive experimental results. VoxDirector demonstrates strong speaker consistency and expressiveness in voice cloning, competitive instruction following in voice design, and good reconstruction performance across different audio domains. The ablation studies also indicate that several proposed training and architectural components contribute to the final performance.
Critical weaknesses
  1. Limited novelty and missing comparisons with closely related work. The proposed framework mainly combines existing techniques, including flow matching, MoE routing, quality conditioning, and curriculum learning. More importantly, closely related works such as ScenA [1], and UNISON [2] are not included in the comparisons, making it difficult to establish the advantages of VoxDirector over existing unified audio generation approaches. [1] Finkelson, M., Segal, D., Richardson, E., et al. (2026). Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors. arXiv:2606.19325. [2] Li, Z., Xu, H., Su, J., et al. (2026). UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion. arXiv:2605.31530.

  2. Insufficient evaluation of the core audio scene generation capabilities. VoxBench-Scene relies on three overall scores: instruction accuracy, acoustic quality, and expressiveness. However, these metrics do not directly assess temporal synchronization, ambience continuity, speaker binding, or cross-source acoustic consistency, which are central motivations of unified audio scene generation. Moreover, the improvements over the cascaded baseline are relatively modest (around 0.2 points on a five-point scale), without statistical significance analysis or human preference comparisons.

  3. Fairness and strength of the baseline comparisons. Dasheng AudioGen does not support voice cloning and is evaluated on a different subset, raising concerns about the comparability of the reported results. Meanwhile, the Qwen3-TTS + TangoFlux baseline relies on relatively simple, rule-based audio mixing rather than an optimized cascaded pipeline. Consequently, the current comparisons are insufficient to convincingly establish the advantages of unified generation over existing alternatives.

Questions for authors

Please see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer jYTH (1/2)

Thank you for pointing out the need for comparisons with related work and more detailed scene evaluations. We apologize that the original submission did not make the comparison scope and evaluation details clear.

1. Scene comparisons

Thank you for suggesting comparisons with ScenA and UNISON.

VoxBench-Scene contains 500 test cases, divided into five categories of 100 according to the presence of dialogue, the number of speakers, and the presence of local sound effects. Five listeners rate each evaluated output on instruction accuracy, acoustic quality, and expressiveness on a 1–5 scale. The table reports the actual number of evaluated cases, n, and mean scores ± 95% confidence intervals. Because the systems differ in their supported tasks, Dasheng is evaluated on cases without reference audio: 100 cases without dialogue and 50 description-only cases in each spoken category. UNISON is included only in scenes without dialogue and single-speaker scenes. ScenA supports reference binding for up to three speakers and evaluates 96 and 94 cases in the two multi-speaker categories.

SceneModelnHuman instruction accuracyHuman qualityHuman expressiveness
No dialogue + local effectsQwen3-TTS + TangoFlux1003.50±0.193.99±0.083.61±0.14
Dasheng AudioGen1003.06±0.123.42±0.203.36±0.07
ScenA1003.37±0.062.92±0.153.63±0.11
UNISON1003.01±0.132.94±0.092.41±0.18
VoxDirector1003.72±0.174.51±0.123.99±0.05
Single speaker, no local effectsQwen3-TTS + TangoFlux1003.34±0.084.10±0.163.56±0.20
Dasheng AudioGen502.51±0.143.07±0.113.13±0.06
ScenA1003.21±0.052.83±0.173.24±0.13
UNISON1002.18±0.102.11±0.142.05±0.19
VoxDirector1003.85±0.164.46±0.073.98±0.12
Single speaker + local effectsQwen3-TTS + TangoFlux1003.22±0.114.13±0.203.51±0.08
Dasheng AudioGen502.80±0.053.39±0.133.24±0.17
ScenA1003.39±0.182.76±0.093.18±0.12
UNISON1002.04±0.112.39±0.062.20±0.15
VoxDirector1003.65±0.074.42±0.143.94±0.19
Multiple speakers, no local effectsQwen3-TTS + TangoFlux1003.37±0.133.95±0.183.76±0.10
Dasheng AudioGen502.15±0.093.01±0.122.93±0.20
ScenA963.35±0.142.98±0.052.59±0.11
VoxDirector1003.70±0.184.49±0.134.08±0.06
Multiple speakers + local effectsQwen3-TTS + TangoFlux1003.32±0.104.07±0.193.65±0.14
Dasheng AudioGen502.62±0.153.44±0.113.33±0.06
ScenA943.49±0.122.91±0.172.66±0.09
VoxDirector1003.67±0.164.52±0.054.10±0.13

All five systems evaluate the same 100 cases without dialogue and with local sound effects, on which VoxDirector has the highest mean human scores on all three dimensions. VoxDirector also has the highest reported means in the other four categories, with each system's evaluated scope indicated by n.

3. Scene properties

Thank you for suggesting separate checks for these four scene capabilities.

We conducted human checks comparing VoxDirector with the Qwen3-TTS + TangoFlux cascade on the four dimensions you proposed, using 100 applicable cases for each:

PropertyVoxDirectorQwen3-TTS + TangoFlux
Correct event order100%100%
Sound-field consistency97%36%
Speaker binding100%93%
Uninterrupted background76%54%

We do not support precise timestamp inputs. The event-order check examines whether the generated audio follows the order specified in the description; both systems score 100%. The sound-field consistency check examines whether speech, ambience, and effects sound as though they occupy the same acoustic space. VoxDirector scores 97%, compared with 36% for the cascade, whose outputs more often exhibit shifts in the sound field.

The speaker-binding check covers both character-to-voice assignment and timbre stability across turns; the two systems score 100% and 93%, respectively. The uninterrupted-background check counts any pause in the background as an interruption, including natural gaps between bird calls. Under this criterion, the proportion with uninterrupted background is 76% for VoxDirector and 54% for the cascade, a difference of 22 percentage points.

AuthorsOpenReview ↗

Response to Reviewer jYTH (2/2)

2. Contributions

Thank you for raising the question of novelty.

Dialogue, ambience, and local sound effects in a complete audio scene share a timeline but change at different scales: character identity must remain stable across turns, expressive delivery and local effects change with the narrative, and the background may span the scene. VoxDirector jointly generates these sounds in a single latent sequence and supports both voice design from text and cloning from reference audio (Section 1). Its core designs are as follows:

  1. Style matrix and hierarchical acoustic conditioning. The VoxCaps style matrix maps character archetypes to audible voice and performance cues, explicitly distinguishing stable character attributes across turns from local delivery states within an utterance. During generation, transcripts are aligned to the latent timeline, while global and character-level descriptions enter the denoising network through cross-attention, allowing scene-spanning acoustic conditions and local delivery to participate jointly in generation (Sections 3.3 and 4.3; Appendix A.3).

  2. Dual-granularity task–acoustic routing. The Task Router selects a voice-design or cloning expert for the entire sample. The Audio Router selects acoustic experts frame by frame, with its routing budget adjusted over denoising stages. Dynamic Top-P selection and null paths allow different frames to use different numbers of acoustic experts; the acoustic branch can be skipped while the task expert remains active. The two routes thus handle sample-wide task conditions and frame-varying sounds separately (Section 4.4; Appendix C.4).

  3. Implicit reinforcement learning through outcome conditioning. We adopt an outcome-conditioned approach to implicit reinforcement learning, converting offline audio-quality evaluations into quality descriptions and a global quality level supplied through cross-attention and AdaLN-Zero, respectively. The model learns audio distributions under different quality conditions through flow-matching training. At inference, specifying high-quality conditions steers generation toward high-quality outputs. Recordings of moderate quality can therefore still contribute expressive styles, while the desired audio quality is explicitly specified as a generation condition (Section 4.3; Appendix C.2; Emmons et al., 2022).

4. Human listening results and confidence intervals

Thank you for suggesting that we examine the performance differences through human evaluation and statistical intervals.

The approximately 0.2-point differences you noted are from the LLM-based scores in the original Table 5. We have now added category-level human listening evaluations on VoxBench-Scene and report 95% confidence intervals.

As shown in Section 1, on the same 100 multi-speaker cases with local sound effects evaluated by both VoxDirector and the cascade, human-rated instruction accuracy is 3.67±0.16 versus 3.32±0.10, acoustic quality is 4.52±0.05 versus 4.07±0.19, and expressiveness is 4.10±0.13 versus 3.65±0.14. The differences are 0.35, 0.45, and 0.45 points, respectively, with non-overlapping 95% confidence intervals on all three dimensions. These results show VoxDirector's advantage over the cascade in human listening evaluation.

5. Cascade setup and fairness

Thank you for asking us to clarify the comparison scope and the cascade setup.

VoxDirector and the cascade evaluate exactly the same 100 cases in every category of VoxBench-Scene. We built the cascade using publicly available components that performed well at the time: Qwen3-TTS 1.7B VoiceDesign and Base handle voice design from descriptions and reference cloning, respectively; VoiceDesign obtains 4.09 MOS in the role-play listening evaluation in Table 4. TangoFlux generates ambience, music, and local effects.

Even when the components and mixing are well tuned, the cascade must synthesize dialogue utterance by utterance, generate background and effects on separate tracks, then align and mix them. Model calls and post-processing costs increase with the number of turns and tracks. Independently generated sounds lack shared acoustic context: a character's timbre may change across utterances, and speech and ambience are less likely to sound as though they occupy the same space. VoxDirector jointly generates the whole scene in a single latent sequence and models these relationships together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.