- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ExpressTTS-Bench, an automated benchmark for scenario-conditioned expressiveness in TTS. It generates test cases through a hierarchical scenario tree across five expression modes, two languages, and four instruction formats, and evaluates generated speech with Gemini-3.6-Flash using 45 expression-mode-specific rubrics over nine emotion, prosody, and paralanguage dimensions. The paper evaluates 6 open- and closed-source TTS systems on 8,000 instruction-instance pairs, reports aggregate and fine-grained results, and presents repeated-judge and human-comparison analyses intended to support the reliability of the proposed LALM-as-a-Judge protocol.
Critical strengths
- The benchmark design has useful breadth. It crosses 2 languages, 4 input styles, and 5 qualitatively distinct expression modes, including long-form expression and natural-agent settings that are often neglected in style-control evaluations.
Critical weaknesses
-
This paper does not demonstrate sufficient novelty at ICLR. The incremental contribution over existing benchmarks is not established. InstructTTSEval is cited and directly motivates the instruction formats used in the benchmark, but the paper provides neither an empirical comparison on shared models nor a sufficiently clear account of the additional evaluation capabilities or diagnostic value offered by ExpressTTS-Bench. The primary contribution is just a new benchmark. It remains difficult to assess whether the proposed scenario taxonomy and rubric design provide substantial benefits beyond existing instruction-following and model-as-a-judge evaluation.
-
The evaluation has substantial validation gaps: 1) The same Gemini-3.6-Flash model is used to generate scenarios, construct rubrics, and score model outputs, creating a tightly coupled generation-evaluation loop with a risk of correlated judge bias. Although the paper includes comparisons with 2 additional judges, both are from the Gemini family, so these comparisons do not adequately rule out shared biases. 2) The human study provides some support for overall preference agreement but does not directly validate the fine-grained automated scores. Humans use a unified holistic criterion covering emotion, prosody, and paralanguage, whereas the judge scores 9 dimensions using 45 expression-mode-by-subdimension rubrics. Thus, the reported 71-72% pairwise agreement does not establish the validity of the individual dimensions or their mode-specific rubrics, which underpin the paper’s fine-grained conclusions.
-
The reliability analysis has important statistical and reporting limitations. 1) Section 4.4 describes Spearman correlations as “within 95% CI,” but Table 2 reports only point estimates, without confidence intervals or an explanation of how they are computed. Although the text states that the statistics are based on model rankings, the aggregation procedure is insufficiently specified. Moreover, consistency of aggregate rankings across 6 systems does not establish stability of sample-level or dimension-level scores. 2) The human consistency score in Table 3 is a leave-one-rater-versus-majority agreement rate. It should be supplemented with inter-rater reliability measures, such as Fleiss’ κ or Krippendorff’s α, and uncertainty estimates. Human-human agreement uses the consensus of the other 3 raters, whereas model-human agreement uses all 4 raters, so the 2 percentages are not directly equivalent. The handling of tied votes and the mapping of model scores to the human “no clear difference” category also need clarification. Without accounting for these differences and reporting uncertainty, the similar percentages provide limited support for the claimed proximity to human internal consistency.
Questions for authors
-
Can you provide human validation of the fine-grained scores on a representative subset, covering at least the three broad categories of emotion, prosody, and paralanguage? Please report confidence intervals for agreement overall and for key language and expression-mode subgroups where sample sizes permit.
-
How are the 800 sampled test cases aggregated into the model rankings used for the Spearman and Kendall statistics in Section 4.4? Please clarify the effective number of ranked observations, the confidence-interval method, and the sampling or temperature settings for each judge. A summary of repeated-judge score differences at the sample and dimension levels would also help distinguish ranking stability from scoring stability.
-
How are tied human votes and “no clear difference” judgments handled in Table 3, and how are model score differences mapped to the same preference categories?
3 comments
Official Comment by Authors (2/3)
Weakness 3: The reliability analysis has important statistical and reporting limitations.
We conduct a consistency analysis experiment using a total of samples. Following EmergentTTS-Eval[2], we perform multiple random partial samplings on the results of these samples and take the 2.5% and 97.5% quantiles as the percentile 95% CI, thereby labeling it as "within 95% CI." We will provide a detailed introduction to the calculation method in the final version to avoid misunderstanding. Similarly, we compute the Spearman correlation on the aggregated rankings of all models, consistent with EmergentTTS-Eval. We further compute the Spearman correlation at the sample level and calculate the average: with Gemini-3.6-Flash as the reference, the average sample-level Spearman correlation for Gemini-3.7-Flash is , and the average sample-level Spearman correlation for Gemini-3.1-Pro-preview is . In addition, we find that the Spearman correlation on the overall ranking between Qwen3-Omni and Gemini-3.6-Flash is , while the average sample-level Spearman correlation is only . We find that the score distribution of Qwen3-Omni is more concentrated compared to Gemini, and on some samples the expressiveness scores of different models are relatively close. Since Spearman uses only rankings and does not consider score gaps, the overall scores across more samples can more stably reflect the consistency performance of the judge model.
Following MINT-Bench[3], we separately calculate the consistency of human internal ratings and the consistency between models and humans. We further calculate inter-rater reliability measures, including Fleiss' and Krippendorff's . The results show Fleiss' = and Krippendorff's = , which means that human evaluation shows clear preferences while also having a certain degree of subjectivity. We will report these results in the final version to enhance understanding of human evaluation of expressiveness.
Regarding consensus statistics, although Human Internal consistency and Model-Human consistency respectively use the opinions of and evaluators, we believe that the majority-vote opinions of multiple human evaluators can represent human consensus. We define the unique highest majority vote as human consensus, and when there is no highest vote for the corresponding sample, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is clearly a tie option is it counted as a human tie. For cases where human consensus is a tie, we also strictly determine whether the model opinion and human consensus are consistent based on whether the expressiveness scores obtained by the two audio clips in the actual automatic evaluation are equal. We will add a more detailed introduction to the experimental setup in the final version to aid understanding.
Question 1: Can you provide human validation of the fine-grained scores on a representative subset, covering at least the three broad categories of emotion, prosody, and paralanguage? Please report confidence intervals for agreement overall and for key language and expression-mode subgroups where sample sizes permit.
As mentioned in the response to Weakness 2, in order to conduct a finer-grained human consistency analysis of our automated evaluation, we further carry out a new round of human evaluation on the same sample set across three categories: emotion, prosody, and paralanguage.
| Consistency | Emotion | Prosody | Paralanguage | Overall |
|---|---|---|---|---|
| Human Internal | 55.32% | 65.48% | 33.84% | 69.96% |
| Model-Human | 48.48% | 50.29% | 48.53% | 67.82% |
We find that the Human Internal consistency and Model-Human consistency on individual subdivided dimension are both significantly lower than the consistency in overall expressiveness. Model-Human consistency in paralanguage is even noticeably higher than Human Internal consistency. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability on individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions. We will report and discuss this part of the results in the final version. Because human evaluation is very expensive and inefficient, we conduct human evaluation experiments only on a limited set of samples. We report point estimates of consistency for overall expressiveness and each attribute category, without statistical confidence intervals.
Official Comment by Authors (3/3)
Question 2: How are the 800 sampled test cases aggregated into the model rankings used for the Spearman and Kendall statistics in Section 4.4? Please clarify the effective number of ranked observations, the confidence-interval method, and the sampling or temperature settings for each judge. A summary of repeated-judge score differences at the sample and dimension levels would also help distinguish ranking stability from scoring stability.
Following EmergentTTS-Eval, we calculate the Spearman correlation and the Kendall's W coefficient across multiple sets of model ranking results to measure the consistency between multiple evaluations. Under this approach, the results show that good consistency is demonstrated both within the same judge model itself and among different LALMs. We conduct observations on the rankings of all six models. Following EmergentTTS-Eval, we perform multiple random partial samplings on the results of these samples and take the 2.5% and 97.5% quantiles as the percentile 95% CI, thereby labeling them as "within 95% CI." Therefore, we will provide a detailed introduction to the calculation method in the final version to avoid misunderstanding. For each judge model, we all set the temperature to 0, but the results returned in actual API calls may still vary.
Question 3: How are tied human votes and “no clear difference” judgments handled in Table 3, and how are model score differences mapped to the same preference categories?
We define the unique highest majority vote as human consensus, and when there is no highest vote count in the corresponding sample, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is explicitly a tie option is it counted as a human tie. We map the expressiveness scores of the two models on the corresponding audio into win/tie/loss according to their relative levels. For cases where the human consensus is a tie, we strictly determine whether the model opinion and the human consensus are consistent based on whether the expressiveness scores obtained by the two audio clips in the actual automatic evaluation are equal. In the final version, we will add a more detailed introduction to the details of the experimental setup to aid understanding.
References
[1] InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems. Huang et al. Arxiv preprint.
[2] EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge. Manku et al. NeurIPS 2025.
[3] MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech. Chen et al. ACM MM 2026.
Official Comment by Authors (1/3)
There are some misunderstandings here. Inspired by InstructTTSEval[1], we introduce diverse instruction formats into the test set and provide an introduction and citation in the paper. However, our work emphasizes evaluating the scenario-based expressiveness of TTS models, whereas InstructTTSEval mainly evaluates the instruction-following ability of models, which is a completely different model capability from scenario-based expressiveness. Therefore, the results of the two can't be directly compared. Our work evaluates completely different model capabilities from InstructTTSEval, and we only reference its idea of constructing multi-format instructions rather than making incremental improvements to it. We will further clarify the differences between the two to avoid misunderstandings.
Thank you for your valuable suggestions. In this work, we mainly use Gemini because it is currently one of the strongest multimodal foundation models. Considering potential model bias, we additionally use Qwen3-Omni as a judge model to conduct a consistency analysis under the same experimental setting as in Table 2(b). The results show that the Spearman correlation between Qwen3-Omni and Gemini-3.6-Flash is 0.714, which also demonstrates good consistency. In addition, because human evaluation is relatively expensive and inefficient, in our human evaluation we evaluate overall expressiveness rather than all nine fine-grained sub-dimensions. To conduct a finer-grained human consistency analysis of our automated evaluation, we further carry out a new round of human evaluation on the same sample set across three categories: emotion, prosody, and paralanguage.
We find that consistency in individual subdivided dimension, whether Human Internal or Model-Human, is significantly lower than consistency in overall expressiveness. Model-Human consistency in paralanguage is even noticeably higher than consistency among humans. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability in individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions. We will report this part of the results in the final version.