- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies an important property of voice design: a single textual specification should permit multiple distinct, instruction-compliant speaker identities. It introduces VoxDICE, with hierarchical voice descriptions and a 10,000-utterance real-speech reference set. Evaluation combines within-condition speaker-embedding diversity (MPCD and Vendi Score), global distribution matching/coverage (FSD and SRS), and instruction-following rate. Human judgments validate local diversity proxies, while a fixed-sample speaker-insertion experiment probes the global metrics. An instruction-gated, collision-based GRPO reward is applied to Qwen3-TTS, improving aggregate diversity and instruction following over its original checkpoint.
Critical strengths
-
The benchmark defines a consequential evaluation problem and studies complementary failure modes with substantial empirical effort. Holding both the instruction and synthesis text fixed while sampling K=10 outputs is an appropriate way to expose conditional repetition. Adding global reference alignment/coverage and instruction adherence prevents any single diversity score from standing in for overall voice-design quality. The 10,000-utterance curation, three instruction scales, seven baseline systems, and GRPO intervention with gate/diversity ablations constitute meaningful benchmarking work. The reversed local/global ordering of VoxCPM2 and Ming-omni-TTS under APS illustrates why these axes are informative. Reusing Emilia and InstructTTSEval templates does not negate the contribution of constructing and validating this evaluation protocol.
-
The paper provides substantive human validation rather than relying solely on embedding metrics or an untested LLM judge. Section 5.4 covers 195 audio groups, 1,950 pairs, and 5,850 judgments from 30 participants. Vendi Score obtains Pearson/Spearman correlations of 0.783/0.791 with perceived pairwise diversity, and MPCD obtains 0.797/0.795. Appendix A.2 separately evaluates 900 instruction-audio cases with three annotations each (2,700 judgments); human-judge agreement is 80.7%, 83.3%, and 72.3% for APS, DSD, and RP. The global-metric experiment fixes the utterance count at 10,000, varies AISHELL-2 speaker count from 20 to 1,991, and repeats sampling 20 times per count. These are material validation efforts and support the metrics as useful diagnostics, with the limits discussed below.
Critical weaknesses
Reproducibility remains a concern. Although the authors promise to release the benchmark and evaluation code upon acceptance, the currently accessible materials do not clearly support reconstructing the exact reference set and evaluation pipeline, making independent reproduction of the reported results uncertain.
The validation supports useful proxies, but does not yet establish the reliability of the benchmark's central comparative claims. The local correlations are computed on 195 groups deliberately stratified into 13 Vendi Score bins. This supports association over a broad selected range, but does not by itself establish human-consistent model rankings or discrimination between nearby systems. The paper does not report inter-annotator reliability, model/scale composition of these groups, or how the 'Unsure' option and three-way splits in the annotation interfaces are mapped to the binary majority labels. Likewise, the 900-case IFR check reports raw agreement without class balance, error breakdowns, or a human-checked comparison of the model differences central to the conclusions. The lower RP agreement warrants particular care; it does not, by itself, prove that a reported IFR difference is spurious. For global metrics, the balanced real-speech speaker-insertion test is useful but does not test robustness to the nonuniform speaker frequencies and acoustic conditions of generated speech. FSD measures reference moment alignment, while SRS extrapolates reference-entry coverage under a uniform-occupancy assumption; neither should be read as a direct count of perceptually distinct speakers. Finally, diversity is measured over all K outputs and IFR separately, leaving the intended quantity—diversity among compliant alternatives—unmeasured. RP illustrates the ambiguity: VS increases from 3.520 to 4.040 while IFR falls from 50.0% to 48.4%. These gaps limit how confidently the benchmark can rank systems for the stated use case, even though they do not invalidate the observed correlations or negate the study's workload.
Questions for authors
-
Artifact-access and reproducibility clarification: Is an anonymous evaluation package already available to reviewers? Please identify what is accessible now versus promised after acceptance. In particular, can the exact reference IDs/segments and three-scale instructions, corpus composition and speaker-duplication audit, CAM++ checkpoint/preprocessing, scoring code, baseline configurations/seeds, and cached judge/annotation outputs be inspected or reconstructed exactly? Please also state the source-data version and reuse terms. This request concerns evaluating the benchmark; it does not require releasing the private RL training corpus.
-
Human-validation clarification and targeted analysis: How were the 195 groups distributed across models and prompt scales, and how were 'Unsure' responses and votes with no majority handled in both human studies? Please report inter-annotator agreement and class balance/confusion information for the IFR check. Do the existing human annotations support the main model ordering, particularly the original-versus-GRPO comparison where represented, with uncertainty computed at the group/case level? A focused reanalysis of existing annotations would be more informative than simply collecting more correlated votes.
-
Global-metric interpretation and robustness: How many distinct or conservatively clustered speakers are represented by the 10,000 reference utterances, and how does residual duplication affect the scores? Can the existing speaker-insertion experiment be reweighted to test unequal speaker frequencies and report whether the conclusions survive? Please distinguish alignment to the selected expressive-speech reference from a general claim about real-world speaker diversity, and clarify the sensitivity of the SRS extrapolation to its uniform-occupancy assumption.
-
Targeted evidence for the benchmark's intended use: Under a fixed generation budget, can you report diversity restricted to instruction-compliant outputs together with compliant-output yield, explicitly handling cases with fewer than two valid outputs? Applying this to the main original-versus-GRPO comparison, with a human-checked subset or documented judge uncertainty, would materially affect my assessment of whether the benchmark measures diverse yet compliant voice choices.
4 comments
Rebuttal by Authors (2/4)
W1 & Q1: Reproducibility and artifact access
An anonymous evaluation package is now available at https://huggingface.co/datasets/ICLR45502/VoxDice_anonymous. It contains everything required to evaluate the benchmark and excludes only the private RL training corpus. We have verified that the released code reproduces our main results. The package includes:
- Reference set. All 10,000 reference utterances are provided as the original Emilia segments, with each file named by its utterance ID, so every segment can be traced to the source data. Each utterance comes with its transcript and its APS, DSD and RP instructions.
- Corpus audits. The corpus composition and the audit of highly similar utterances (Appendices A.2–A.3) can be reconstructed from the released audio and CAM++ embeddings.
- CAM++. We use the public checkpoint https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common. The preprocessing code fixes the resampling, the FBank front end and the embedding normalization.
- Scoring code. The code computes every diversity metric, including V-MPCD. It also includes the LLM-judge script with its model, prompt and decoding settings.
- Generation code. We provide generation code for all eight systems. It uses a deterministic seed scheme: each utterance receives its own seed, derived from a fixed base seed and logged together with the output.
The cached judge outputs and human annotations are not in the repository because of their volume. They correspond one to one with the generated audio, and the released judge script can regenerate them. All reference audio comes from the Chinese subset of the original Emilia dataset, which is distributed under the CC BY-NC license, and VoxDICE follows the same non-commercial terms. Gemini-3.1-Pro-Preview generated the APS instructions and GPT-4o generated the DSD and RP instructions (Section 3.2.2, pages 3–4). The templates are given in Appendix A.14. The trained GRPO checkpoints will be released upon acceptance.
W2 & Q2: Human-consistent ranking and discrimination between nearby systems
We agree that a correlation computed on groups stratified across a wide range of Vendi Scores establishes association, but it does not show that the metrics can discriminate between nearby systems. We therefore added a two-alternative forced-choice (2AFC) study that measures how large a score difference listeners can perceive (Section 5.5, Figure 5, page 10, lines 508–527; Appendix A.12).
- Trial design. Each trial presents two groups of ten utterances generated by two different models from the same instruction and synthesis text. The annotator selects the group with more distinct speakers, which mirrors how systems are compared in practice.
- Tier matching. We split the case-level scores of all eight systems at their tertiles into Low, Mid and High tiers. We pair only groups that fall in the same tier under both VS and MPCD.
- Sampling. Within each tier we sample 100 pairs from each of five |ΔVS| intervals, giving 1,500 pairs annotated by 30 participants. For MPCD, the same pairs are re-binned by |ΔMPCD|.
The table reports the win rate of the higher-scoring group as VS / MPCD.
| Score difference (VS / MPCD) | Low tier | Mid tier | High tier |
|---|---|---|---|
| [0, 0.2) / [0, 0.02) | 56.0% / 55.2% | 48.0% / 46.9% | 51.0% / 53.0% |
| [0.2, 0.4) / [0.02, 0.04) | 74.0%*** / 72.2%*** | 57.0% / 53.1% | 48.0% / 46.4% |
| [0.4, 0.6) / [0.04, 0.06) | 82.0%*** / 74.2%*** | 70.0%*** / 72.5%*** | 57.0% / 58.4% |
| [0.6, 0.8) / [0.06, 0.08) | 96.0%*** / 93.3%*** | 81.0%*** / 82.1%*** | 69.0%*** / 70.2%** |
| ≥ 0.8 / ≥ 0.08 | 98.0%*** / 98.0%*** | 89.0%*** / 86.9%*** | 82.0%*** / 85.0%*** |
** p < 0.01 and *** p < 0.001, one-sided z test against 50% with Holm correction.
These results lead to three conclusions:
- Rankings agree with listeners once the gap is large enough. Above a modest score gap, listeners consistently agree with the metric about which system is more diverse. In every tier, once the gap exceeds the smallest intervals, the win rate rises steadily with the gap and reaches 82–98% in the largest interval.
- The resolution limit for nearby systems is now explicit. No tier departs from chance in the smallest interval, so differences of that size should not be read as perceptual. The perceptibility threshold grows with the tier, from about 0.2 VS or 0.02 MPCD in the Low tier to roughly three times that in the High tier.
- The reported improvement is perceptible. Qwen3-TTS lies in the Low and Mid tiers. GRPO raises its average VS by 0.541 and its average MPCD by 0.061, and both gains exceed the thresholds of these tiers, so the main original-versus-GRPO ordering is audible to listeners.
Rebuttal by Authors (3/4)
W2 & Q2: Annotation details and reanalysis of existing annotations
The revision reports all of the requested information.
- Composition of the 195 groups. Appendix A.11 and Table 16 list the number of groups per system and scale (pages 24–25, lines 1286–1307). Groups are sampled uniformly within the VS bins, so each system's share follows its VS distribution. Every system contributes 17 to 36 groups.
- "Unsure" and no-majority votes. Both studies use a strict rule, and no case is left undecided.
- In the timbre study, a pair is labeled "same speaker" only when "Same" votes outnumber "Different" votes. "Unsure" counts toward neither side (Appendix A.13.1, page 26, lines 1357–1360).
- In the IFR check, a case is labeled "follows" only when "Matches" votes outnumber "Does not match" votes (Appendix A.4, page 18, lines 961–966).
- In both studies, all remaining cases receive the negative label.
- Inter-annotator reliability. Fleiss' κ is 0.73 for the timbre study (Section 5.4, page 9, line 478). For the IFR check it is 0.72, 0.75 and 0.67 on APS, DSD and RP (Appendix A.4, page 18, lines 967–971). For the new effective-voice study it is 0.781, 0.762 and 0.675 (Table 15).
- Class balance and confusion for the IFR check. Table 8 reports the full TP/FP/TN/FN counts together with the precision, recall and NPV of the judge (page 19, lines 972–1006).
- The human-positive rate is 60.3%, 69.0% and 50.7% on APS, DSD and RP.
- The chance-corrected judge–human agreement (Cohen's κ, computed from Table 8) is 0.59, 0.58 and 0.45.
Do the existing annotations support the original-versus-GRPO comparison? We examined this question and found that the existing annotations are too few, and not paired, to support a statistical test of this comparison.
- In the 195-group study, Qwen3-TTS and GRPO contribute only 17 and 19 groups, spread across three scales and not matched by prompt.
- In the IFR check, each model has 29 to 44 unpaired cases per scale. With about 40 binary labels per arm, the standard error of an IFR difference is about 9–10 percentage points, which far exceeds the 3–5 point differences at stake.
We agree with the reviewer that collecting more correlated votes would not help. We therefore ran the targeted effective-voice study described above. It is paired at the prompt level and uses 100 prompts per scale. For each prompt and model, the number of effective voices and the compliance rate are averaged over the three annotators, and the 100 prompts of each scale are treated as the independent units of analysis. The study shows with statistical significance on all three scales that GRPO produces more compliant and perceptually distinct candidate voices.
W2 & Q3: Global metrics under unequal speaker frequencies and the uniform-occupancy assumption
We reweighted the speaker-insertion experiment as the reviewer suggested (Section 3.3.3, Figure 2, page 5, lines 251–269). We sample 100 to 1,991 AISHELL-2 speakers and fix the total at 10,000 utterances. Utterances are allocated by a Zipf law with exponent s ∈ {0, 0.4, 0.6, 0.8}, where s = 0 is the balanced setting, and each setting is repeated 20 times. The conclusions hold under every allocation.
- Monotonicity in speaker count. Both metrics remain strongly monotonic in the number of speakers. For s = 0, 0.4, 0.6 and 0.8, Spearman ρ is 0.954, 0.951, 0.947 and 0.935 for SRS and −0.878, −0.869, −0.864 and −0.837 for FSD, with all p < 10⁻⁵⁰.
- Sensitivity to skew. At a fixed speaker count, a more uneven allocation lowers SRS and raises FSD, and mean SRS is ordered s = 0 > 0.4 > 0.6 > 0.8 at every speaker count.
A skewed allocation concentrates speech on a few dominant timbres and lowers effective diversity, so both metrics capture how evenly utterances are spread as well as how many speakers are present.
Sensitivity to the uniform-occupancy assumption. With 10,000 generated samples, the extrapolation changes the score very little. Averaged over 20 runs, the largest gap D̂ − k between the extrapolated support D̂ and the observed number of distinct retrieved entries k is 29.7 entries, which is 0.30% of the reference. This occurs at s = 0 with 1,960 speakers, where k = 2,276.5 and D̂ = 2,306.2. The reported SRS is therefore driven mainly by the observed retrieval coverage. The assumption also errs in the conservative direction. For a given support, uniform sampling yields the most distinct entries. A skewed generator repeats entries more often, which pushes k and D̂ downward rather than inflating them, so uneven generation is penalized, consistent with the lower SRS at larger s. The revision states that neither FSD nor SRS is a direct count of perceptually distinct speakers (Section 3.3.2, page 4, lines 205–207). It also states that the insertion experiment establishes monotonic sensitivity rather than a calibrated mapping to speaker counts (Section 3.3.3, page 5, line 269).
Rebuttal by Authors (4/4)
Q3: Distinct speakers in the reference and the effect of residual duplication
Clustering threshold. We count timbre groups with a conservative criterion calibrated on AISHELL-2, which provides speaker labels. With CAM++ embeddings, utterances of the same speaker have a mean cosine similarity of 0.710 (s.d. 0.101). The most similar cross-speaker utterance pair, measured over 200 random speakers, has a mean similarity of only 0.549 (s.d. 0.093). A threshold of 0.7 therefore groups only voices that are acoustically very close.
Group counts. Average-linkage clustering of the 10,000 reference embeddings at this threshold yields 9,555 timbre groups (Appendix A.3, Table 5, page 17, lines 865–888).
| Number of groups | Size 1 | Size 2 | Size 3 | Size 4 | Size 5 |
|---|---|---|---|---|---|
| 9,555 | 9,224 | 245 | 63 | 18 | 5 |
Only 7.8% of the utterances share a group with another utterance, and no group has more than five members. Emilia provides no speaker labels, so we cannot determine whether a highly similar pair comes from two distinct speakers with close voices or from the same speaker appearing in different recordings. In either case, such pairs reflect the non-uniform density of the empirical timbre distribution, so we keep them in the main evaluation (Appendix A.3, page 17, lines 890–897; Section 3.2.1, page 3, lines 152–155).
Effect on the scores. To measure the effect of residual duplication, we built 20 density-flattened references. Each keeps one utterance per group and refills the set with filtered Emilia utterances, preserving the size and the gender and age composition, so that no pair exceeds the threshold. We then recomputed FSD and SRS with the generated embeddings fixed (Table 6, pages 17–18, lines 898–933).
- FSD. FSD changes by less than 0.13% for every model.
- Self-retrieval ceiling. The ceiling drops from 0.823 to 0.814 because leave-one-out retrieval becomes harder. This raises SRS by a nearly uniform 1.14–1.16 percentage points for every model.
- Generated coverage. Coverage decreases by 0.23–1.46 points for all models except Qwen3-TTS-GRPO, which gains 0.19. Generated speech, like the reference itself, retrieves fewer distinct entries once retrieval becomes harder, which confirms that the leave-one-out ceiling is an appropriate normalizer.
- Rankings. The model ranking under both FSD and SRS is unchanged.
Residual duplication therefore does not affect any comparative conclusion.
Q3: Alignment to the selected reference versus real-world speaker diversity
We agree with this distinction, and the revision adopts it. The paper does not claim coverage of real-world speaker diversity in general. The reference is constructed as an empirical reference distribution of the speech variation observed in Emilia (Section 3.2.1, page 3, lines 137–152). It contains in-the-wild recordings with at most one segment per source recording, is balanced between male and female speakers with an additional subset of child speech, and is filtered to reduce the dominance of flat, neutral speech (Appendix A.2, page 16, lines 811–863). A lower FSD or a higher SRS therefore means, strictly, closer alignment with or broader coverage of this constructed reference. The FSD and SRS gains of GRPO are described in the same terms (Appendix A.8, page 22, lines 1184–1186).
This alignment is nevertheless informative about diversity. In the insertion experiment above, the evaluated sets differ only in how many speakers they contain and how evenly utterances are allocated. Against the same fixed reference, sets with more speakers and a more even allocation consistently obtain a lower FSD and a higher SRS. The AISHELL-2 speech is read speech that was not used to build the reference, so these trends reflect the number and allocation of speakers rather than a match to the reference's speaking style. More diverse distributions therefore achieve closer similarity to the reference.
We hope these responses address the reviewer's concerns and would be glad to provide further clarification.
Rebuttal by Authors (1/4)
We thank the reviewer for the careful and constructive assessment. We respond to each concern below. All section, page and line numbers refer to the revised manuscript.
W2 & Q4: Diversity among instruction-compliant outputs under a fixed budget
We agree that reporting diversity over all K outputs separately from IFR cannot show whether added diversity comes from compliant alternatives or from instruction violations. The revision measures diversity within the compliant subset in two independent ways. The first is automatic and accounts for the uncertainty of the judge. The second is a direct human count.
Valid MPCD. For each instruction and synthesis text we generate a fixed budget of K = 10 utterances. Let M be the number of utterances that the judge labels as compliant, which we call the compliant-output yield. Its expectation is K × IFR. V-MPCD is the mean pairwise cosine distance among these M utterances. When M ≤ 1, V-MPCD is set to zero and the case is kept in the average. Diversity contributed by non-compliant outputs is therefore ignored, and cases with low yield are penalized directly (Section 3.3.1, Eq. 2, page 4, lines 181–186). V-MPCD is computed on the same 1,000 cases for every model (Table 1, page 8). Models with weaker instruction following lose more diversity when moving from MPCD to V-MPCD, and RP shows the largest drop for every model. This confirms the ambiguity that the reviewer identified. For the main comparison, GRPO raises V-MPCD on every scale. This includes RP, where IFR decreases slightly, so the gain cannot be explained by additional instruction violations (Section 5.3, page 9, lines 449–452).
Judge uncertainty. V-MPCD depends on the judge's labels, so we propagate the judge's errors into it with a two-stage bootstrap (Appendix A.5, pages 19–20, lines 1008–1042).
Judge errors shift the absolute values because flipped labels change each valid set. On every scale, however, GRPO keeps a higher V-MPCD and the 95% intervals of the two models do not overlap. This holds even on RP, where the judge agrees least with humans.
Human count of effective voices. We also ran a new study that relies on neither the judge nor the speaker encoder (Appendix A.10, Table 15, pages 23–24, lines 1231–1284).
GRPO yields 10–12% more effective voices on every scale. On RP, the number of effective voices rises significantly while human-judged compliance is essentially unchanged, which resolves the RP ambiguity that the reviewer raised. Taken together, the automatic and human results indicate that the benchmark captures diverse yet compliant voice choices. They also indicate that GRPO increases diversity among compliant outputs, not only among all outputs.