acceptodds
Under review as a conference paper at ICLR 2027

ExpressTTS-Bench: Benchmarking the Scenario-based Expressiveness of Text-to-Speech Models

Abstract

Recent Text-to-Speech (TTS) systems have achieved substantial progress in sound quality and integrity, yet their ability to generate natural and appropriate highly expressive audio based on a given scenario is still very limited. Meanwhile, the existing evaluation of expressiveness still relies on costly and inefficient human evaluation. Moreover, existing expressiveness evaluation typically overlooks the scenario and textual context that fundamentally shape how expressiveness should be perceived. To this end, we introduce ExpressTTS-Bench, a fully automated benchmark for evaluating the scenario-based expressiveness of TTS models. ExpressTTS-Bench constructs diverse test data through a hierarchical scenario tree covering five representative expression modes and four task formats, resulting in test samples across English and Chinese. To enable fine-grained and scalable evaluation, we decompose expressiveness into nine sub-dimensions spanning emotion, prosody, and paralanguage, and introduce a LALM-as-a-Judge framework with automatically generated, expression-mode-specific scoring rubrics. We evaluate six representative open-source and closed-source TTS models and reveal substantial differences in scenario-based expressiveness across models, languages, instruction formats, expression modes, and sub-dimensions. Consistency analyses and human evaluation further support the reliability of ExpressTTS-Bench. ExpressTTS-Bench provides a scalable framework for systematically evaluating the scenario-conditioned expressive capabilities of modern TTS systems.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Speech, Expressiveness, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p3hDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ExpressTTS-Bench, an automated benchmark for scenario-conditioned expressiveness in TTS. It generates test cases through a hierarchical scenario tree across five expression modes, two languages, and four instruction formats, and evaluates generated speech with Gemini-3.6-Flash using 45 expression-mode-specific rubrics over nine emotion, prosody, and paralanguage dimensions. The paper evaluates 6 open- and closed-source TTS systems on 8,000 instruction-instance pairs, reports aggregate and fine-grained results, and presents repeated-judge and human-comparison analyses intended to support the reliability of the proposed LALM-as-a-Judge protocol.

Critical strengths
  • The benchmark design has useful breadth. It crosses 2 languages, 4 input styles, and 5 qualitatively distinct expression modes, including long-form expression and natural-agent settings that are often neglected in style-control evaluations.
Critical weaknesses
  • This paper does not demonstrate sufficient novelty at ICLR. The incremental contribution over existing benchmarks is not established. InstructTTSEval is cited and directly motivates the instruction formats used in the benchmark, but the paper provides neither an empirical comparison on shared models nor a sufficiently clear account of the additional evaluation capabilities or diagnostic value offered by ExpressTTS-Bench. The primary contribution is just a new benchmark. It remains difficult to assess whether the proposed scenario taxonomy and rubric design provide substantial benefits beyond existing instruction-following and model-as-a-judge evaluation.

  • The evaluation has substantial validation gaps: 1) The same Gemini-3.6-Flash model is used to generate scenarios, construct rubrics, and score model outputs, creating a tightly coupled generation-evaluation loop with a risk of correlated judge bias. Although the paper includes comparisons with 2 additional judges, both are from the Gemini family, so these comparisons do not adequately rule out shared biases. 2) The human study provides some support for overall preference agreement but does not directly validate the fine-grained automated scores. Humans use a unified holistic criterion covering emotion, prosody, and paralanguage, whereas the judge scores 9 dimensions using 45 expression-mode-by-subdimension rubrics. Thus, the reported 71-72% pairwise agreement does not establish the validity of the individual dimensions or their mode-specific rubrics, which underpin the paper’s fine-grained conclusions.

  • The reliability analysis has important statistical and reporting limitations. 1) Section 4.4 describes Spearman correlations as “within 95% CI,” but Table 2 reports only point estimates, without confidence intervals or an explanation of how they are computed. Although the text states that the statistics are based on model rankings, the aggregation procedure is insufficiently specified. Moreover, consistency of aggregate rankings across 6 systems does not establish stability of sample-level or dimension-level scores. 2) The human consistency score in Table 3 is a leave-one-rater-versus-majority agreement rate. It should be supplemented with inter-rater reliability measures, such as Fleiss’ κ or Krippendorff’s α, and uncertainty estimates. Human-human agreement uses the consensus of the other 3 raters, whereas model-human agreement uses all 4 raters, so the 2 percentages are not directly equivalent. The handling of tied votes and the mapping of model scores to the human “no clear difference” category also need clarification. Without accounting for these differences and reporting uncertainty, the similar percentages provide limited support for the claimed proximity to human internal consistency.

Questions for authors
  • Can you provide human validation of the fine-grained scores on a representative subset, covering at least the three broad categories of emotion, prosody, and paralanguage? Please report confidence intervals for agreement overall and for key language and expression-mode subgroups where sample sizes permit.

  • How are the 800 sampled test cases aggregated into the model rankings used for the Spearman and Kendall statistics in Section 4.4? Please clarify the effective number of ranked observations, the confidence-interval method, and the sampling or temperature settings for each judge. A summary of repeated-judge score differences at the sample and dimension levels would also help distinguish ranking stability from scoring stability.

  • How are tied human votes and “no clear difference” judgments handled in Table 3, and how are model score differences mapped to the same preference categories?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Weakness 1: This paper does not demonstrate sufficient novelty at ICLR. The incremental contribution over existing benchmarks is not established. InstructTTSEval is cited and directly motivates the instruction formats used in the benchmark, but the paper provides neither an empirical comparison on shared models nor a sufficiently clear account of the additional evaluation capabilities or diagnostic value offered by ExpressTTS-Bench. The primary contribution is just a new benchmark. It remains difficult to assess whether the proposed scenario taxonomy and rubric design provide substantial benefits beyond existing instruction-following and model-as-a-judge evaluation.

There are some misunderstandings here. Inspired by InstructTTSEval[1], we introduce diverse instruction formats into the test set and provide an introduction and citation in the paper. However, our work emphasizes evaluating the scenario-based expressiveness of TTS models, whereas InstructTTSEval mainly evaluates the instruction-following ability of models, which is a completely different model capability from scenario-based expressiveness. Therefore, the results of the two can't be directly compared. Our work evaluates completely different model capabilities from InstructTTSEval, and we only reference its idea of constructing multi-format instructions rather than making incremental improvements to it. We will further clarify the differences between the two to avoid misunderstandings.

Weakness 2: The evaluation has substantial validation gaps.

Thank you for your valuable suggestions. In this work, we mainly use Gemini because it is currently one of the strongest multimodal foundation models. Considering potential model bias, we additionally use Qwen3-Omni as a judge model to conduct a consistency analysis under the same experimental setting as in Table 2(b). The results show that the Spearman correlation between Qwen3-Omni and Gemini-3.6-Flash is 0.7140.714, which also demonstrates good consistency. In addition, because human evaluation is relatively expensive and inefficient, in our human evaluation we evaluate overall expressiveness rather than all nine fine-grained sub-dimensions. To conduct a finer-grained human consistency analysis of our automated evaluation, we further carry out a new round of human evaluation on the same sample set across three categories: emotion, prosody, and paralanguage.

ConsistencyEmotionProsodyParalanguageOverall
Human Internal55.32%65.48%33.84%69.96%
Model-Human48.48%50.29%48.53%67.82%

We find that consistency in individual subdivided dimension, whether Human Internal or Model-Human, is significantly lower than consistency in overall expressiveness. Model-Human consistency in paralanguage is even noticeably higher than consistency among humans. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability in individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions. We will report this part of the results in the final version.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Weakness 3: The reliability analysis has important statistical and reporting limitations.

We conduct a consistency analysis experiment using a total of 800800 samples. Following EmergentTTS-Eval[2], we perform multiple random partial samplings on the results of these samples and take the 2.5% and 97.5% quantiles as the percentile 95% CI, thereby labeling it as "within 95% CI." We will provide a detailed introduction to the calculation method in the final version to avoid misunderstanding. Similarly, we compute the Spearman correlation on the aggregated rankings of all models, consistent with EmergentTTS-Eval. We further compute the Spearman correlation at the sample level and calculate the average: with Gemini-3.6-Flash as the reference, the average sample-level Spearman correlation for Gemini-3.7-Flash is 0.490.49, and the average sample-level Spearman correlation for Gemini-3.1-Pro-preview is 0.410.41. In addition, we find that the Spearman correlation on the overall ranking between Qwen3-Omni and Gemini-3.6-Flash is 0.710.71, while the average sample-level Spearman correlation is only 0.070.07. We find that the score distribution of Qwen3-Omni is more concentrated compared to Gemini, and on some samples the expressiveness scores of different models are relatively close. Since Spearman uses only rankings and does not consider score gaps, the overall scores across more samples can more stably reflect the consistency performance of the judge model.

Following MINT-Bench[3], we separately calculate the consistency of human internal ratings and the consistency between models and humans. We further calculate inter-rater reliability measures, including Fleiss' \upkappa\upkappa and Krippendorff's \upalpha\upalpha. The results show Fleiss' \upkappa\upkappa = 0.24770.2477 and Krippendorff's \upalpha\upalpha = 0.24870.2487, which means that human evaluation shows clear preferences while also having a certain degree of subjectivity. We will report these results in the final version to enhance understanding of human evaluation of expressiveness.

Regarding consensus statistics, although Human Internal consistency and Model-Human consistency respectively use the opinions of 4−1=34 - 1 = 3 and 44 evaluators, we believe that the majority-vote opinions of multiple human evaluators can represent human consensus. We define the unique highest majority vote as human consensus, and when there is no highest vote for the corresponding sample, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is clearly a tie option is it counted as a human tie. For cases where human consensus is a tie, we also strictly determine whether the model opinion and human consensus are consistent based on whether the expressiveness scores obtained by the two audio clips in the actual automatic evaluation are equal. We will add a more detailed introduction to the experimental setup in the final version to aid understanding.

Question 1: Can you provide human validation of the fine-grained scores on a representative subset, covering at least the three broad categories of emotion, prosody, and paralanguage? Please report confidence intervals for agreement overall and for key language and expression-mode subgroups where sample sizes permit.

As mentioned in the response to Weakness 2, in order to conduct a finer-grained human consistency analysis of our automated evaluation, we further carry out a new round of human evaluation on the same sample set across three categories: emotion, prosody, and paralanguage.

ConsistencyEmotionProsodyParalanguageOverall
Human Internal55.32%65.48%33.84%69.96%
Model-Human48.48%50.29%48.53%67.82%

We find that the Human Internal consistency and Model-Human consistency on individual subdivided dimension are both significantly lower than the consistency in overall expressiveness. Model-Human consistency in paralanguage is even noticeably higher than Human Internal consistency. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability on individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions. We will report and discuss this part of the results in the final version. Because human evaluation is very expensive and inefficient, we conduct human evaluation experiments only on a limited set of 200200 samples. We report point estimates of consistency for overall expressiveness and each attribute category, without statistical confidence intervals.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Question 2: How are the 800 sampled test cases aggregated into the model rankings used for the Spearman and Kendall statistics in Section 4.4? Please clarify the effective number of ranked observations, the confidence-interval method, and the sampling or temperature settings for each judge. A summary of repeated-judge score differences at the sample and dimension levels would also help distinguish ranking stability from scoring stability.

Following EmergentTTS-Eval, we calculate the Spearman correlation and the Kendall's W coefficient across multiple sets of model ranking results to measure the consistency between multiple evaluations. Under this approach, the results show that good consistency is demonstrated both within the same judge model itself and among different LALMs. We conduct observations on the rankings of all six models. Following EmergentTTS-Eval, we perform multiple random partial samplings on the results of these samples and take the 2.5% and 97.5% quantiles as the percentile 95% CI, thereby labeling them as "within 95% CI." Therefore, we will provide a detailed introduction to the calculation method in the final version to avoid misunderstanding. For each judge model, we all set the temperature to 0, but the results returned in actual API calls may still vary.

Question 3: How are tied human votes and “no clear difference” judgments handled in Table 3, and how are model score differences mapped to the same preference categories?

We define the unique highest majority vote as human consensus, and when there is no highest vote count in the corresponding sample, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is explicitly a tie option is it counted as a human tie. We map the expressiveness scores of the two models on the corresponding audio into win/tie/loss according to their relative levels. For cases where the human consensus is a tie, we strictly determine whether the model opinion and the human consensus are consistent based on whether the expressiveness scores obtained by the two audio clips in the actual automatic evaluation are equal. In the final version, we will add a more detailed introduction to the details of the experimental setup to aid understanding.

References

[1] InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems. Huang et al. Arxiv preprint.

[2] EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge. Manku et al. NeurIPS 2025.

[3] MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech. Chen et al. ACM MM 2026.

Reviewer_QeMDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ExpressTTS-Bench evaluates scenario-conditioned expressiveness in TTS using 8,000 test cases across English and Chinese, five expression modes, and four instruction formats. A hierarchical LLM-based pipeline constructs scenarios and synthesis instructions. The authors use 30 natural reference recordings per expression mode to derive 45 mode/dimension-specific rubrics, then use Gemini-3.6-Flash to score nine dimensions covering emotion, prosody, and paralanguage. Their average forms the overall expressiveness score. Six TTS systems are evaluated, revealing differences across languages, instruction formats, expression modes, and dimensions. Validation includes repeated judging, comparison among three Gemini judges, and 200 human-rated audio pairs, with reported model-human agreement of 72% for English and 71% for Chinese.

Critical strengths
  1. The benchmark addresses a useful gap in TTS evaluation: the same acoustic behavior can be effective or inappropriate depending on the intended situation. The five expression modes, two languages, and four instruction formats provide a broad and structured set of tests beyond generic speech quality. The per-mode and per-dimension results are more informative than a single leaderboard; for example, the aggregate results can obscure substantial differences in long-form expression (Figure 3). This scope and diagnostic organization make the dataset potentially useful to researchers developing controllable TTS systems.

  2. The paper provides a concrete, scalable evaluation design and some encouraging validation. The separation of emotion, prosody, and paralanguage, and the use of explicit mode-specific rubrics, make the intended measurements inspectable. The appendix includes detailed prompts and human-evaluation criteria, and the authors state that the full dataset, code, and rubric set are supplied as supplementary materials. The four-expert listening study is a meaningful attempt to test perceptual relevance: the reported 71-72% model-human agreement is reasonably close to the approximately 74% leave-one-out human agreement. These results suggest useful signal, although they do not yet establish all the stronger validity claims.

Critical weaknesses
  1. The interpretation of the headline score as better scenario-conditioned expressiveness is not sufficiently justified. Section 3.3 averages four degree-axis dimensions and five quality-axis dimensions with equal weight (Table 17). However, Appendix F deliberately defines degree independently of appropriateness: the emotional-intensity rubric allows a contextually absurd performance to receive a high intensity score (p.37), and the paralanguage-richness rubric does not penalize excessive vocal events merely because they are ill-suited to the text (p.40). This is sensible for descriptive diagnostics, but averaging these quantities into a higher-is-better overall score permits additional expressive activity to compensate for poorer contextual fit. Calibration to an expression mode does not resolve differences between situations within that mode. Moreover, the human criteria explicitly state that stronger emotion and more paralanguage are not necessarily better, and that appropriate restraint or absence can be preferable (Appendix C.2). The paper does not show that its equal-weight aggregation captures these trade-offs or that the headline rankings are robust to them. This also weakens the inference in Section 4.3 that lower degree-axis scores necessarily indicate an expressive deficiency.

  2. The validation does not isolate the benefit of the proposed automatic rubric construction or establish the reliability of its fine-grained scores. The 200-pair human study tests holistic preferences, whereas the method makes claims about nine dimensions and 45 mode/dimension combinations. Tables 2a/2b measure agreement of rankings across only six TTS systems, rather than agreement on individual clips or dimensions; additionally, the alternative LALM judges are all Gemini variants. These checks are useful but leave room for systematic shared errors. The reported experiments also do not compare the reference-derived, mode-specific rubrics with a simple common rubric using the same judge and the same human-labeled pairs. Consequently, it remains unclear whether the proposed rubric-generation procedure improves human alignment over a simpler evaluator. Table 3's proximity to human internal agreement is encouraging, but the two rates use different consensus constructions, and uncertainty and tie handling need to be specified before interpreting them as near-human reliability.

Overall, I lean toward weak rejection because the benchmark's central measurement and the added value of its evaluation procedure require stronger validation. The coverage and implementation are promising, and targeted analyses of the existing outputs and listening-study pairs could materially improve my assessment.

Post-rebuttal update (19 Nov 2026):

The added ablations address my main concerns: combining degree and quality scores agrees better with human preferences than either alone, and the mode-specific rubrics improve overall agreement over a common rubric on the evaluated pairs. Together with the benchmark's coverage, this is enough for me to support weak acceptance, so I have raised my rating from 2 to 3. The final paper should clarify the evaluation rounds, retained sample counts and uncertainty, and avoid claiming that the fine-grained scores are fully validated. These limitations remain, but they no longer outweigh the contribution for me. My final recommendation is weak acceptance (3).

Questions for authors
  1. Aggregation validity: How do the equal-weight degree/quality trade-offs relate to human preferences, particularly for situations requiring restraint? On the existing human-labeled pairs, please compare the full mean with quality-axis-only scoring and report sensitivity of model rankings and human agreement to the degree-axis weight. Please examine disagreements where extra intensity or paralanguage raises the automatic score despite a human preference for a more restrained rendition.

  2. Added value of the evaluator: On the same labeled audio pairs and with the same judge, does using reference-derived, mode-specific rubrics improve agreement over a simple common rubric? A control that removes or mismatches the scenario context would also help test the claimed benefit of scenario conditioning. Please distinguish these contributions from the benefit of using a capable LALM judge at all.

  3. Human-validation protocol: Please report the exact counts by language and expression mode, the handling of ties and cases without a majority among the four annotators, and how scalar model-score differences are mapped to win/tie/loss. Can you provide confidence intervals and a comparable, preferably paired, analysis of human-model versus human-human agreement? Please also clarify which fine-grained conclusions are supported by direct human validation versus only aggregate model-ranking stability.

4 comments
Reviewer_QeMDOpenReview ↗

Note on AI tool use during my review

Following the PCs' request, I'm adding a note about how AI tools were used during my reviewing process.

I discussed this paper with ChatGPT (GPT-6-Astra) while working on my review. I used it as a sounding board: I shared my own reading notes — mainly the aggregation/weighting concern around Table 17, the inconsistency between the rubrics in Appendix F and the human criteria in Appendix C.2, and the limitations of the reported agreement statistics — and asked it to check whether my criticisms were technically sound and whether I had overlooked obvious counterarguments. It also helped me polish the wording of the review text.

My assessment of the paper, the rating, and all specific claims in the review are my own: I verified every cited detail (the rubric contents, Table 17, the 71-72% vs ~74% agreement numbers, etc.) directly against the manuscript. The AI discussion helped me articulate the aggregation concern more precisely, but it did not change my overall assessment. My questions for the authors in the review stand as written, and I look forward to their responses.

AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Weakness 1: The interpretation of the headline score as better scenario-conditioned expressiveness is not sufficiently justified.

Thanks for your valuable review and suggestions. We classify all nine sub-dimensions into two categories: degree-axis and quality-axis. To further verify the necessity of degree-axis attributes and quality-axis attributes, we use the average scores of degree-axis attributes and quality-axis attributes at different proportions as the final expressiveness scores, and calculate consistency with human evaluation in the same way as in Section 4.4. The results are as follows:

AttributesAllENZH
Full71.50%72.00%71.00%
Degree-axis only64.50%63.00%66.00%
Quality-axis only39.50%45.00%34.00%
Degree/Quality (50%/50%)72.00%71.00%73.00%

The results show that when using only the degree-axis or the quality-axis dimension, the consistency between model evaluation and human evaluation is significantly lower than when using all dimensions, while the 50%/50% equal-weight result is close to the original result, indicating that considering both the degree-axis and quality-axis dimension information in evaluation is helpful and necessary. Therefore, we merge all nine sub-dimensions into a scenario-based expressiveness score, thereby achieving balanced and comprehensive evaluation across all dimensions. On the one hand, score aggregation appears to allow extra expressive activity to compensate for poorer scenario fit, but on the other hand, a lower scenario fit score will greatly penalize unreasonable extra expressive activity and thus lower the total score. For real TTS systems, the two evaluate different target dimensions rather than being in a mutually compensatory relationship. In addition, the evaluation results in Section 4.4 show good consistency with human evaluation results, which further verifies the effectiveness of our evaluation pipeline.

Weakness 2: The validation does not isolate the benefit of the proposed automatic rubric construction or establish the reliability of its fine-grained scores.

Thank you for your valuable suggestions! We further conduct a new round of human evaluation on the same sample set across three categories: emotion, prosody, and paralanguage:

ConsistencyEmotionProsodyParalanguageOverall
Human Internal55.32%65.48%33.84%69.96%
Model-Human48.48%50.29%48.53%67.82%

We find that the consistency in individual subdivided aspects, both Human Internal and Model-Human, is significantly lower than the consistency in overall expressiveness. Model-Human consistency in paralanguage is even noticeably higher than Human Internal consistency. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability in individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions.

In this work, we mainly use Gemini because it is currently one of the strongest multimodal foundation models. To further strengthen the conclusion, we additionally used Qwen3-Omni as a judge model to conduct experiments on the same sample set, and the results show that the Spearman correlation between Qwen3-Omni and Gemini-3.6-Flash is 0.7140.714, which also shows good consistency.

To verify the effectiveness of exclusive rubric generation, we conduct ablation experiments on the same sample set as human evaluation under two settings: general rubrics and no rubrics. All prompts include the necessary scenario and character background information. For the target sub-dimensions, no_rubrics shows only the sub-dimension names in the prompt, while general_rubrics provide each target sub-dimension with a simple general description applicable to all expression modes. The consistency results between evaluations using different rubrics and human assessment are as follows:

ConsistencyEmotionProsodyParalanguageOverall
Full Rubrics48.48%50.29%48.53%67.82%
No Rubrics47.27%52.05%41.91%63.21%
General Rubrics45.45%55.56%44.85%62.64%

We find that using automatically constructed rubrics based on specific expression modes and specific sub-dimensions achieves the highest Model-Human consistency, which demonstrates the effectiveness of our rubrics. In addition, we find that using general rubrics even performs worse compared to no_rubrics. We speculate that this may be because Gemini, as a powerful multimodal understanding foundation model, is instead distracted by using the same rubrics across differentiated scenarios.

We will incorporate all these results and analyses into our final version. Thank you again for your review and suggestions!

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Question 1: Aggregation validity: How do the equal-weight degree/quality trade-offs relate to human preferences, particularly for situations requiring restraint?

As reported in the response to Weakness 1, we use the average scores of the degree-axis dimensions and the quality-axis dimensions at different proportions as the final expressiveness scores, and measure consistency with human evaluation in the same way as in Section 4.4. The results show that when only the degree-axis or quality-axis dimensions are used, the consistency between model evaluation and human evaluation is significantly lower than when all dimensions are considered, while the 50%/50% equal-weight result is close to the Original result, indicating that considering both degree-axis and quality-axis dimension information in evaluation is helpful and necessary. We will report these analysis results of degree-axis and quality-axis trade-off in the final version to further enhance understanding of the impact of the degree-axis and quality-axis dimensions on overall expressiveness.

Question 2: Added value of the evaluator: On the same labeled audio pairs and with the same judge, does using reference-derived, mode-specific rubrics improve agreement over a simple common rubric?

As mentioned in our answer to Weakness 2, we conduct ablation experiments on the same sample set as the human evaluation under two settings: general rubrics and no rubrics. We find that using automatically constructed rubrics based on specific expression modes and specific sub-dimensions achieves the best consistency performance, which demonstrates the effectiveness of our rubrics. In addition, we conduct further analysis and discussion based on the results of the ablation experiments. We will report all these valuable results in our final version.

Question 3: Human-validation protocol: Please report the exact counts by language and expression mode, the handling of ties and cases without a majority among the four annotators, and how scalar model-score differences are mapped to win/tie/loss. Can you provide confidence intervals and a comparable, preferably paired, analysis of human-model versus human-human agreement? Please also clarify which fine-grained conclusions are supported by direct human validation versus only aggregate model-ranking stability.

We randomly select 200200 samples from the eight test subsets for human consistency analysis, including 100100 in Chinese and 100100 in English. Counted by test subset as:

Subsetn
EN-AL / EN-Direct / EN-NL / EN-SPG34 / 22 / 25 / 19
ZH-AL / ZH-Direct / ZH-NL / ZH-SPG31 / 20 / 25 / 24

Counted by expression mode: empathetic_companionship 4949, information_broadcast 3636, long_form_expression 3939, natural_agent 2929, role_expression 4747.

We define the unique majority highest vote as human consensus, and when the corresponding sample doesn't have a highest vote, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is explicitly a tie option is it counted as a human tie. We map the expressiveness scores of the two models on the corresponding audio into win/tie/loss according to their relative levels. For cases where the human consensus is a tie, we strictly determine whether the model opinion and the human consensus are consistent based on whether the expressiveness scores obtained by the two audio clips in the actual automatic evaluation are equal. Because human evaluation is expensive and inefficient, we conduct only one round of human evaluation on the sampled target set. Additionally, based on your suggestion, we conduct a new round of human evaluation on the same sample set across the three categories of emotion, prosody, and paralanguage.

ConsistencyEmotionProsodyParalanguageOverall
Human Internal55.32%65.48%33.84%69.96%
Model-Human48.48%50.29%48.53%67.82%

We find that the consistency in individual subdivided dimension, both Human Internal and Model-Human, is significantly lower than the consistency in overall expressiveness. The consistency between Model-Human in paralanguage is even noticeably higher than Human Internal. At the same time, both of them show good and similar consistency in overall expressiveness. We find that human evaluators have limited evaluation stability in individual fine-grained dimensions. Therefore, this further highlights the effectiveness of comprehensive expressiveness evaluation across all dimensions. We will report and discuss this part of the results in the final version.

AuthorsOpenReview ↗

Thank you so much !

Thank you for recognizing our paper and raising the score. We are very glad that our responses have addressed your concerns. We are very grateful for your review and valuable suggestions, which can help us further improve our work. We will include the new results and analyses in our final version, and further clarify the specific experimental setup and boundaries to enhance rigour and reproducibility. We sincerely hope our work can inspire and help the community.

Reviewer_vRAcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ExpressTTS-Bench, an automated benchmark for whether TTS models produce expressive speech that fits a given scenario. Gemini-3.6-Flash generates the test data through a hierarchical scenario tree, covering five expression modes and four instruction formats, for 8,000 English and Chinese samples. Expressiveness is split into nine sub-dimensions over emotion, prosody and paralanguage. For each mode and sub-dimension, the same Gemini model derives a rubric from 30 reference clips and then scores the generated audio. Six voice-design TTS models are evaluated, with consistency checks and a small pairwise human study.

Critical strengths
  • Conditioning expressiveness evaluation on scenario and text is a useful and timely direction.
  • The instruction formats appear to share the same situation and target text, so format differences can be attributed to the format itself.
  • The rubrics are thoughtful: separating degree-type from quality-type dimensions avoids simply rewarding more exaggerated speech, and all prompts are included.
  • Some findings are useful for developers, such as weak paralanguage across models and instability in long-form generation.
Critical weaknesses
  1. The judge's reliability is not established.

Gemini is among the strongest audio understanding models available, so using it as the judge is reasonable. However, the paper does not establish where its judgments can be trusted.

  • Self-consistency and cross-judge results use Spearman correlations over rankings of only six models, providing limited statistical evidence and no sample-level agreement.
  • All three judges are Gemini models and may share the same blind spots.
  • The human study is small (200 pairs, four raters), covers only overall preference, and reports neither chance-corrected agreement nor how ties are handled.
  • Most analyses are per-dimension, but no individual dimension is validated against humans.

The evidence does not sufficiently support the claim of "strong agreement with human judgments."

  1. It is unclear whether the judge can reliably detect paralanguage.

In Appendix C.1, the Paralanguage Richness evaluation states that the clip has no audible breaths, while Appropriateness praises its "low-profile breath resets." The two Dynamic Naturalness evaluations also describe the same clip in contradictory ways.

Since a main finding is that all models score near the floor on paralanguage, this may partly reflect limitations of the judge.

Possible validation experiments include:

  • Per-dimension human ratings on a subset of samples.
  • Controlled probes that insert or remove breaths and laughter, or modify speech rate and pitch range.
  • Evaluation on real speech with annotated non-verbal events.
  1. Data, rubrics, and judge come from the same model, with limited quality control.

Generating test text with an LLM is reasonable, but the rubrics are derived from Gemini's own perception of reference audio. The rubric generator and judge may therefore share the same blind spots.

Reference clips are described in only one sentence. Human checking is mentioned only in the LLM-use statement, without sampling or rejection rates.

Rubric generation is also a claimed contribution, but no ablation is provided. The paper should compare against generic rubrics or no rubrics in terms of human agreement, and test sensitivity to different reference clips.

  1. Some evaluation design choices may introduce artifacts.
  • Direct format: The TTS model receives no instruction, while the judge still sees the character profile. The prosody rubric asks whether pitch matches the character's age and social standing.
  • SPG instructions: Every instruction ends with a fixed invitation to include sighs, laughter, or sobs, even for information broadcasting.
  • Equal weighting: Two paralanguage dimensions remain near the floor for every model, while Appropriateness can award full marks even when no paralanguage is present. These dimensions may contribute little or have conflicting effects on aggregate scores.
  1. Model coverage is narrow, and uncertainty estimates are missing.

Only voice-design models are evaluated. General instruction-following TTS and end-to-end speech models are excluded. It is also unclear whether native expressive controls, such as non-verbal tags, were used.

Table 1 provides no confidence intervals, despite small differences between leading models and several format comparisons.

  1. The positioning overstates the gap in prior work.

Several closely related works are missing:

  • Chiang et al. (Findings of EMNLP 2025) evaluate speaking style and role-play speech using audio LLM judges, with human comparisons.
  • VStyle covers role-play and implicit empathy.
  • AudioJudge (EACL 2026) investigates limitations of audio judges on fine-grained paralinguistics.
  • NV-Bench targets non-verbal vocalizations in TTS.

None is cited. EmergentTTS-Eval and InstructTTSEval are cited but not compared against.

Minor Concerns

  • The mapping from 1–10 judge scores to Table 1 is unclear.
  • Table 17 appears without Tables 4–16.
  • The phrase "within 95% CI" is ambiguous.
  • The number of excluded ElevenLabs samples is not reported.
  • The paper lacks a limitations section.
Questions for authors
  1. Can you report sample-level, per-dimension repeat agreement (e.g. ICC across runs), per-dimension human agreement (at least for paralanguage), a chance-corrected statistic for the human study, and how ties are handled?
  2. Where do the reference clips come from, were the rubrics reviewed by people, and how stable are rubrics and rankings with a different set of clips? How much generated data was checked by hand and removed?
  3. In the Direct format, how are voice-design models run without a voice description, and does the judge still apply voice- and age-related criteria?
  4. Were native expressive controls (e.g. non-verbal tags) used for ElevenLabs v3 and Fish Audio?
6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/6)

Weakness 1: The judge's reliability is not established.

Thank you for your valuable review and suggestions. Following EmergentTTS-Eval[1], we calculate the Spearman correlations and the Kendall's W coefficients of the rankings of multiple models to measure the consistency across multiple evaluations. The results indicate that good consistency is demonstrated both within the same judge model and among different LALMs. In addition, we also report sample-level consistency statistics. For the three pairs of runs, we calculate the average Spearman correlation of model rankings across all samples, and the results are as follows:

Round PairSpearman
Original - Repeat 10.501
Original - Repeat 20.537
Repeat 1 - Repeat 20.610

This shows that multiple evaluations based on the same judge model still have good consistency on individual samples. However, it can also be found that the average consistency at the sample level is significantly lower than the consistency of the overall ranking on all data. We notice that on some samples, the expressiveness scores of different models are relatively close. Since Spearman only uses ranking and doesn't consider score gaps, the overall scores across more samples can more stably reflect the consistency performance of the judge model.

To further strengthen the conclusion, we additionally use Qwen3-Omni as a judge model to conduct experiments on the same set. The results show that the Spearman correlation between Qwen3-Omni and Gemini-3.6-Flash is 0.7140.714, which also demonstrates good consistency.

Given the limited resources, we hire four expert annotators to conduct independent evaluations on 200200 sample pairs. We report the consistency results within human annotations and between model and human in Table 3 of the paper, and find that the consistency between the model and human consensus can approach that within human evaluators. We define the unique majority vote as the human consensus, and when there is no highest vote for the corresponding sample, we define it as an invalid no-consensus sample and discard it. Only when the majority vote is explicitly a tie option is it counted as a human tie. For cases where the human consensus is a tie, we strictly determine whether the model opinion and the human consensus are consistent based on whether the expressiveness scores obtained by the two audios in the actual automatic evaluation are equal. We will add a more detailed introduction in our final version to aid understanding.

Weakness 2: The reliability of judge models in paralanguage.

There may be some misunderstanding here. In the example, the evidences provided by the model when evaluating paralanguage richness and appropriateness are not contradictory. When evaluating richness, the model takes the degree of expression as the evaluation target, and "...... are cleanly gated without perceptible respiratory resets, ......" mainly emphasizes that the degree of expression is very weak. When evaluating appropriateness, the description "clean, low-profile breath resets" also corresponds to the weak expression mentioned above. In addition, many works such as NVV-SuperBench[2] has also used frontier LALMs as paralanguage judges and verified their effectiveness. Therefore, considering that human evaluation on a large number of samples is expensive and difficult to reproduce, we select frontier audio understanding models as the judge model for paralanguage evaluation. At the same time, we also strongly agree that current LALMs still have limitations in acoustic perception, including paralanguage recognition. As the capability of judge models improves, the quality of evaluation will also be further enhanced. Thank you again for your suggestion. We will further clarify and discuss this limitation in our final version.

AuthorsOpenReview ↗

Official Comment by Authors (2/6)

Weakness 3: Data, rubrics, and judge come from the same model, with limited quality control.

Thank you for your valuable insights and suggestions. Considering that human annotation is expensive and poorly scalable, we try to provide evaluations of the model's performance in diverse scenarios without relying on human annotation. Therefore, in this paper, we built efficient automated test data synthesis, rubric construction, and audio evaluation pipelines based on LLMs. We mainly use Gemini in this work because it is currently one of the strongest multimodal foundation models. For automated rubric generation, our design intent is to transform the target behaviors observed in reference audio clips into interpretable, mode-specific evaluation criteria without relying on a large amount of external manually annotated data. Considering that LALM as a rubric generator and evaluation model may have blind spots, we have also conducted a comprehensive consistency analysis of LALM-as-a-Judge. The results in Table 2 and Table 3 show that the judge model has good stability and good consistency with human evaluation, which further verifies the effectiveness of our method.

Considering potential model bias, we also additionally use Qwen3-Omni as a judge model to conduct a consistency analysis under the same experimental setting as in Table 2(b). The results show that the Spearman correlation between Qwen3-Omni and Gemini-3.6-Flash is 0.7140.714, which also demonstrates good consistency.

Furthermore, based on your suggestion, we conduct ablation studies on the same sample set as the human evaluation under two settings: general rubrics and no rubrics. All prompts include the necessary scene and character background information. For the target sub-dimension, no rubrics only show the attribute name in the prompt, while general rubrics provide each target sub-dimension with a simple general description applicable to all expression modes. The consistency results between evaluations using different rubrics and human evaluation are as follows:

ConsistencyEmotionProsodyParalanguageOverall
Full Rubrics48.48%50.29%48.53%67.82%
No Rubrics47.27%52.05%41.91%63.21%
General Rubrics45.45%55.56%44.85%62.64%

We find that using automatically constructed rubrics based on specific expression modes and specific sub-dimensions achieves the highest Model-Human consistency, which demonstrates the effectiveness of our rubrics. In addition, we find that using simple general rubrics performs even worse compared to no_rubrics. We speculate that this may be because Gemini, as a powerful multimodal understanding foundation model, is instead distracted by using the same rubrics across differentiated scenarios. We will incorporate these results and analyses into the final version.

AuthorsOpenReview ↗

Official Comment by Authors (3/6)

Weakness 4: Some evaluation design choices may introduce artifacts.

There may be some misunderstandings here. In the Direct subset, the TTS model directly performs audio generation without relying on instructions. This design is intended to evaluate the model's ability to understand and interpret the target scenario based solely on the target text. According to Section 3.1, during dataset generation, the character profile and the specific event description are generated together and are highly correlated with each other, so the character information is also an important component of the target scenario. The target text is further generated based on these information. Therefore, the model needs to infer these information as much as possible when seeing only the target text. Then, during evaluation, we provide all of this reference information to the judge model to assess the model's performance. Since different models have varying degrees of support for paralanguage features, we don't use explicit paralanguage labels for control in the test set. The instructions in the SPG subset use open-ended fixed descriptions for paralanguage, with the aim of evaluating the model's ability to spontaneously incorporate natural paralanguage information based on its understanding of the target scenario. We have provided the complete test dataset in the supplementary materials for reference.

The three sub-dimensions related to paralanguage are all very important for effectively evaluating a model's expressiveness. On the two degree-axis attributes of Richness and Diversity, although all models show significant limitations, the results in Figure 2 and Figure 4 show that different models can produce performance differences of 10+ at maximum. Therefore, evaluation on these two attributes remains important. In addition, it is entirely reasonable for an audio clip with a lower degree of paralanguage expression to receive a high score on the Appropriateness dimension because of its appropriateness of expression. The results in Figure 2 show that each model also has clear performance differences in paralanguage appropriateness. To further verify the necessity of both the degree-axis attributes and the quality-axis attributes, we use the average scores of the degree-axis attributes and the average scores of the quality-axis attributes in different proportions as the final expressiveness scores, and calculate their consistency with human evaluation in the same way as in Section 4.4. The results are as follows:

AttributesAllENZH
Original71.50%72.00%71.00%
Degree-axis only64.50%63.00%66.00%
Quality-axis only39.50%45.00%34.00%
Degree/Quality (50%/50%)72.00%71.00%73.00%

The results show that when using only the degree-axis or the quality-axis dimensions, the consistency between model evaluation and human evaluation is significantly lower than when using all dimensions. The 50%/50% equal-weight result is close to the Original result, indicating that it is helpful and necessary to consider both the degree-axis and quality-axis dimension information in evaluation.

Weakness 5: Model coverage is narrow, and uncertainty estimates are missing.

In this paper, our main goal is to evaluate the ability of TTS models to generate highly expressive audio based on a given scene context. We primarily select currently representative frontier voice design models for evaluation, which allow receiving detailed instructions for audio generation. In addition, although some other general instruction-following TTS models also allow receiving instructions (such as Gemini-TTS, etc.), these non-voice design models require selecting a specified timbre and do not allow receiving character profile information. Considering the consistency of the experimental setup, we don't select this type of model. We also acknowledge that we don't clearly explain the scope of our research in the paper, which may cause misunderstanding. In the final version, we will more accurately define and introduce the scope of our research and conclusions.

To improve the stability of the results, we evaluate each model on a total of 8,0008,000 test samples. Some models perform similarly on certain subsets, and we faithfully report these results. On the one hand, weaker models still show a clear performance gap compared with the leading models; on the other hand, the high consistency between the rankings based on our evaluation results and human evaluations further validates the effectiveness of our evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.