- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
The paper proposes VoxDirector, which is a flow-matching DiT that generates multi-speaker speech with diverse acoustic backgrounds. The system supports voice design from a structured description and voice cloning from reference audio. Its components include VoxCaps (a media-derived data pipeline with a persona library and human Best-Worst scaling), VoxVAE (25-Hz continuous 48-kHz latents), quality-metadata conditioning, a dual-router MoE, and a four-stage curriculum. Evaluation covers a wide range of tasks, from reconstruction, cloning, voice design and full scenes.
Critical strengths
- The work is good as a system design: one model for design, cloning, dialogue and non-speech audio. The Environment / Speakers / Content format cleanly separates stable traits from local delivery.
- Detailed appendices and well-motivated targeted synthesis are plus.
Critical weaknesses
- The contribution is a large system with limited isolated evidence for each component. The main generative ablations (Table 5) are all on one in-house benchmark scored by one LLM judge. It is hard to tell which design choices matter and why. For ICLR, I would expect clearer evidence of what is learned further beyond “this system works well.”
- Claims exceed the results: (1) VoxDirector does not win baselines on InstructTTSEval RP and DSD, despite the paper’s role-play motivation. (2) VoxCPM2 is missing from VoxBench-Instrut though highlighted ahead. (3) The Table 3 baseline numbers are copied from another report, which is okay unless the same judge are used, which would need double checks.(4) Content error is higher than that of TTS-focused systems, which deserves additional explanation.
- The most important scene evaluation is thin and not clear. Basically, voxbench-scene has only two weak baselines with LLM judge, which is also in-house only without other chance to fully understand the performance gain.
minor concerns:
- Apart from VoxDirector itself, the named parts are largely combinations of existing tools:
- VoxCaps connects off-the-shelf models: UVR, 3D-Speaker, Fun-ASR, Qwen3-ForcedAligner, Qwen3.5-Omni, SQUIM, and CosyVoice 3.
- VoxVAE combines SAME-style objectives, EnCodec-style augmentation, and standard discriminators.
- VoxBench-Instruct and VoxBench-Scene are small in-house test sets.
None of these is evaluated or released as an independent artifact, and VoxCaps is not compared against simpler annotation pipelines. Naming each one implies a standalone contribution the paper does not establish. They should be described as components of the system.
- The model is conditioned on SQUIM-derived quality and then evaluated with SQUIM-PESQ. This could introduce a potential issue that the fidelity evaluation may be gamed.
- Expressiveness and all ablations are scored by LLM judges. Judge–human agreement is a Spearman correlation of 0.64–0.72, measured on only 100 clips. No confidence intervals, significance tests, or seeds are reported, and several ablation gaps are small. All above indicate a over-reliance on LLM judge. A bit more information with human checks would be helpful here.
Questions for authors
Covered in the weakness.
6 comments
Response to Reviewer TNCP (2/3)
2. VoxCPM2 comparison and human evaluation of role-play
Thank you for your attention to role-play evaluation and for suggesting the VoxCPM2 comparison.
Some role instructions in InstructTTSEval allow multiple valid interpretations, and its coverage of emotions and character types is uneven. For example, “explain a problem in a teacher's voice” can reasonably be performed with either warmth and patience or seriousness and firmness. Evaluators may interpret such instructions differently, while a simple true/false judgment cannot fully capture subtle differences in delivery. Human listening evaluation is therefore also needed to assess role-play expressiveness.
The benchmark's instructions also do not cover joint control of multi-speaker dialogue and environmental sound effects. VoxDirector is highly competitive on InstructTTSEval, achieving the highest Chinese and English APS scores among the systems in Table 3.
Table 3 follows InstructTTSEval's Gemini 2.5 Pro judging protocol, with APS, DSD, and RP calculated according to the benchmark definitions. The evaluation method is described in Appendix D.3.
VoxBench-Instruct includes advertising, short-drama, and general scenarios. It evaluates role-play through expressiveness, prosodic naturalness, audio fullness, and scene appropriateness. The test set contains 180 cases, with five listeners rating each evaluated output. The complete results after adding VoxCPM2 are as follows:
| Model | Mean MOS | Expressiveness | Prosodic naturalness | Audio fullness | Scene appropriateness |
|---|---|---|---|---|---|
| MiMo-Audio | 3.21 | 3.06 | 3.27 | 3.57 | 2.94 |
| MOSS-VoiceGenerator | 3.60 | 3.42 | 3.62 | 4.06 | 3.31 |
| VoxCPM2 | 3.95 | 3.83 | 3.93 | 4.30 | 3.73 |
| Qwen3-TTS-VD | 4.09 | 3.96 | 4.09 | 4.45 | 3.88 |
| VoxDirector | 4.22 | 4.12 | 4.20 | 4.47 | 4.10 |
VoxDirector has the highest overall human mean and the highest scores on all four individual dimensions, clearly outperforming VoxCPM2. On VoxBench-Scene, which covers multi-speaker speech, ambience, and local sound effects, VoxDirector also achieves the highest reported means on all three human-rated dimensions in all five categories.
3. Content accuracy and multi-task generation
Thank you for your attention to content accuracy. However, characterizing VoxDirector's content accuracy as lower than that of specialized TTS systems is not accurate.
For example, on SwanBench-Speech (Table 2), VoxDirector's single-speaker Error is 0.086, below CosyVoice-2 (0.106) and ZipVoice (0.213); its multi-speaker Error is 0.120, below FireRedTTS-2 (0.148) and VibeVoice (0.204). On Seed-TTS-Eval (Appendix E.3), its Chinese CER is 1.21%, below F5-TTS (1.53%); its English WER is 2.34%, below MaskGCT (2.62%) and CosyVoice 2 (3.09%).
Beyond content accuracy, VoxDirector supports both instruction-driven voice design (Instruct) and zero-shot voice cloning (Zero-shot), covering single-speaker and multi-speaker speech as well as the joint generation of speech, ambience, and sound effects. We therefore cover a much broader range of generation tasks and scenarios. Regarding expressiveness, timbre consistency, expressive richness, and expressive hierarchy all achieve the highest reported values in their respective groups in the voice-cloning evaluation (Table 2).
5. SQUIM in training and independent quality evaluation
Thank you for your attention to the independence of quality evaluation.
During training, SQUIM provides quality estimates in offline data preprocessing, which are used to construct discrete quality descriptions and a global level. Implicit reinforcement learning follows an outcome-conditioning mechanism, with the generator learning audio distributions under the corresponding conditions. The training objective remains flow matching; we neither backpropagate SQUIM gradients through generated audio nor maximize SQUIM rewards through online rollouts.
During evaluation, SQUIM-PESQ corresponds only to the Fidelity column in Table 2. PESQ and STOI in Table 1 and Appendix E.2 use reference-based calculations. Timbre similarity, SRMR, ASR, SpeechJudge, expressiveness evaluation, and human listening evaluations do not rely on SQUIM.
Across the five scene categories evaluated by both VoxDirector and the cascade, VoxDirector's mean human-rated quality ranges from 4.42 to 4.52, compared with 3.95 to 4.13 for the cascade, an improvement of 0.29 to 0.54 points in each category.
Response to Reviewer TNCP (3/3)
4. Human evaluation and scene comparisons
Thank you for suggesting additional human listening evaluations and scene baselines.
We added ScenA and UNISON as unified generation baselines alongside Dasheng AudioGen and the Qwen3-TTS 1.7B + TangoFlux cascade. VoxBench-Scene contains 500 cases in five categories of 100 each. The no-dialogue category has no references; each of the other four categories contains 50 description-only cases and 50 reference-conditioned cases. Five listeners rate each evaluated output, and results are reported as means ± 95% confidence intervals:
| Scene | Model | n | Human instruction accuracy | Human quality | Human expressiveness |
|---|---|---|---|---|---|
| No dialogue + local effects | Qwen3-TTS + TangoFlux | 100 | 3.50±0.19 | 3.99±0.08 | 3.61±0.14 |
| Dasheng AudioGen | 100 | 3.06±0.12 | 3.42±0.20 | 3.36±0.07 | |
| ScenA | 100 | 3.37±0.06 | 2.92±0.15 | 3.63±0.11 | |
| UNISON | 100 | 3.01±0.13 | 2.94±0.09 | 2.41±0.18 | |
| VoxDirector | 100 | 3.72±0.17 | 4.51±0.12 | 3.99±0.05 | |
| Single speaker, no local effects | Qwen3-TTS + TangoFlux | 100 | 3.34±0.08 | 4.10±0.16 | 3.56±0.20 |
| Dasheng AudioGen | 50 | 2.51±0.14 | 3.07±0.11 | 3.13±0.06 | |
| ScenA | 100 | 3.21±0.05 | 2.83±0.17 | 3.24±0.13 | |
| UNISON | 100 | 2.18±0.10 | 2.11±0.14 | 2.05±0.19 | |
| VoxDirector | 100 | 3.85±0.16 | 4.46±0.07 | 3.98±0.12 | |
| Single speaker + local effects | Qwen3-TTS + TangoFlux | 100 | 3.22±0.11 | 4.13±0.20 | 3.51±0.08 |
| Dasheng AudioGen | 50 | 2.80±0.05 | 3.39±0.13 | 3.24±0.17 | |
| ScenA | 100 | 3.39±0.18 | 2.76±0.09 | 3.18±0.12 | |
| UNISON | 100 | 2.04±0.11 | 2.39±0.06 | 2.20±0.15 | |
| VoxDirector | 100 | 3.65±0.07 | 4.42±0.14 | 3.94±0.19 | |
| Multiple speakers, no local effects | Qwen3-TTS + TangoFlux | 100 | 3.37±0.13 | 3.95±0.18 | 3.76±0.10 |
| Dasheng AudioGen | 50 | 2.15±0.09 | 3.01±0.12 | 2.93±0.20 | |
| ScenA | 96 | 3.35±0.14 | 2.98±0.05 | 2.59±0.11 | |
| VoxDirector | 100 | 3.70±0.18 | 4.49±0.13 | 4.08±0.06 | |
| Multiple speakers + local effects | Qwen3-TTS + TangoFlux | 100 | 3.32±0.10 | 4.07±0.19 | 3.65±0.14 |
| Dasheng AudioGen | 50 | 2.62±0.15 | 3.44±0.11 | 3.33±0.06 | |
| ScenA | 94 | 3.49±0.12 | 2.91±0.17 | 2.66±0.09 | |
| VoxDirector | 100 | 3.67±0.16 | 4.52±0.05 | 4.10±0.13 |
n is the actual number of evaluated cases. VoxDirector and the cascade cover all cases; Dasheng evaluates cases without references, UNISON is included only in no-dialogue and single-speaker scenes, and ScenA binds at most three reference speakers.
VoxDirector has the highest reported mean on all three human-rated dimensions in all five categories. On the same 100 multi-speaker cases with local effects evaluated by both systems, it exceeds the cascade by 0.35, 0.45, and 0.45 points in instruction accuracy, acoustic quality, and expressiveness, respectively, with non-overlapping corresponding 95% confidence intervals. In a separate sound-field consistency check on 100 applicable cases, VoxDirector scores 97% and the cascade 36%.
The original Table 4 already contains direct human listening evaluations of expressiveness and three other dimensions, while the VoxVAE ablations in Appendix E.2 use objective reconstruction metrics. Thus, the statement that expressiveness and all ablations are evaluated by an LLM does not match the experimental setup in the paper. The added VoxCPM2 and scene listening evaluations further extend the cross-system comparisons. The paper also reports results on external benchmarks including InstructTTSEval and Seed-TTS-Eval.
6. Data authorization, consent for voice use, and safeguards
Thank you for your attention to source-audio licensing and authorization for voice use.
The source audio and voice uses involved in this study have the relevant authorization and consent. Materials are used within the scope of signed agreements. Public corpora follow their release licenses and original-subcorpus terms. For example, the VoxBox repository uses CC BY-NC-SA 4.0, with subcorpora used under their respective terms.
Custom test reference audio is collected independently and is not added to the training set; participants in subjective listening evaluations provide informed consent. The public VoxCaps repository provides data-processing code, hierarchical prompts, and annotation specifications. The use and redistribution of source audio remain subject to the applicable licenses and agreements.
The Ethics Statement explicitly discusses the risks of voice impersonation and misleading synthetic content and advocates obtaining permission for voice use, disclosing synthetic content, and protecting privacy and intellectual property.
Response to Reviewer TNCP (1/3)
Thank you for your comments. In this response, we add human listening evaluations for VoxCPM2 and a five-system scene evaluation that includes ScenA and UNISON. We address your concerns below.
1. Method contributions, component evaluation, and what the model learns
Thank you for your interest in the method's novelty and the effectiveness of its components.
We study complete audio-scene generation at four levels (data supervision, acoustic representation, the generative model, and evaluation benchmarks) through VoxCaps, VoxVAE, VoxDirector, and VoxBench, respectively.
Compared with annotating an entire recording with a single overall description, VoxCaps labels persistent ambience, stable character attributes, and utterance-level emotions and sound effects separately, using a character-archetype style matrix to organize voice and performance cues. The matrix covers character types in advertising, animation, and short drama, with descriptions grounded in the actual audio. The four-stage processing code, hierarchical prompts, and style matrix are available in the VoxCaps repository (Section 3.3; Appendix A.3).
Building on this structured supervision, VoxDirector introduces two core designs for unified scene generation:
VoxVAE has objective evaluations and ablations independent of the downstream scene generator. Table 1 evaluates speech, general-audio, and music reconstruction on VCTK, FSD50K, and MUSDB18-HQ, respectively; Appendix E.2 ablates its training objectives. Removing the acoustic readout lowers reference-based PESQ from 4.168 to 3.768 and increases MCD from 0.964 to 1.332.
VoxBench contributes defined evaluation tasks for role-play and complete audio-scene generation. VoxBench-Instruct covers advertising, short-drama, and general scenarios, evaluating expressiveness, prosodic naturalness, audio fullness, and scene appropriateness. VoxBench-Scene organizes cases by non-speech content, speaker count, local effects, and reference-voice conditions, comparing instruction following, acoustic quality, and expressiveness within scenes. The two benchmarks assess character performance and multi-source scene generation, respectively; their composition and scoring protocols are described in Appendices D.3 and D.4.