- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HEAR, a training-free framework that assesses the human-likeness of generated speech through acoustic reasoning. HEAR employs off-the-shelf tools to extract acoustic measurements and organizes them into structured semantic evidence at the global, word, and segment levels. An AudioLLM then evaluates the original audio together with the target text and acoustic evidence, yielding an overall human-likeness score, eight dimension-specific scores, and corresponding explanations. The authors also build Humanlike-100, a human-annotated evaluation dataset, and show stronger correlations with human judgments across five AudioLLM backbones. Further experiments on speech preference and perceptual-quality benchmarks confirm the effectiveness of the proposed framework.
Critical strengths
HEAR presents a training-free approach that integrates direct audio perception with structured acoustic evidence at the global, word, and segment levels. This design enables more fine-grained and interpretable human-likeness evaluation without the need for additional model training. It also strengthens correlation with human judgments across five AudioLLM backbones, with especially notable gains on Qwen3-Omni-Thinking. Ablation studies confirm the effectiveness of structured acoustic evidence, while further evaluations on speech preference and naturalness benchmarks support the broader applicability of the framework.
Critical weaknesses
-
Humanlike-100 comprises only 100 speech samples, yet the paper offers no detailed analysis of how emotional expressions, speaking styles, or textual content are distributed. Such information is essential for judging whether the dataset adequately captures diverse speech scenarios and whether the reported improvements generalize to different types of generated speech.
-
Although the paper reports tool-removal and additional-feature experiments, it lacks controlled comparisons that use alternative tools to measure the same acoustic properties. Moreover, the framework instructs the AudioLLM to prioritize the original audio when acoustic evidence conflicts with what it hears, but this behavior is not experimentally validated. Further experiments involving tool substitution and deliberately incorrect or conflicting acoustic tags would help establish the robustness and reliability of the proposed framework.
Questions for authors
Please refer to the Weakness above.
2 comments
Response Part 2/2: Tool Substitution and Conflicting Evidence (Weakness 2)
Thank you for suggesting these controls. We examine whether HEAR is sensitive to the choice of MFCC extractor and how its scores respond to deliberately conflicting tags.
Tool substitution. We compared Librosa and Torchaudio/Kaldi configurations for extracting 13 MFCC coefficients. The configurations use different frame and Mel-filter settings. We aligned the scales of the resulting measurements using 60 separate, unlabeled clips. Audio, target text, prompt, and non-MFCC evidence were held fixed, and each utterance score averages three ratings. Table R2 shows how the swap changes Pearson and Spearman agreement with listeners for each judge.
Table R2. MFCC substitution on Humanlike-100
| MFCC comparison | Librosa r / ρ | Torchaudio/Kaldi r / ρ | Δr [95% CI] | Δρ [95% CI] |
|---|---|---|---|---|
| Qwen2.5-Omni | 0.168 / 0.137 | 0.253 / 0.282 | +0.084 [−0.091, 0.267] | +0.144 [−0.031, 0.323] |
| Gemini-3-Flash Preview | 0.279 / 0.330 | 0.319 / 0.356 | +0.040 [−0.026, 0.114] | +0.026 [−0.045, 0.100] |
| Qwen3-Omni-Thinking | 0.299 / 0.336 | 0.340 / 0.364 | +0.041 [−0.103, 0.200] | +0.028 [−0.106, 0.168] |
As shown in Table R2, replacing the Librosa extractor with the Torchaudio/Kaldi configuration produces small changes in the correlations. The mean absolute difference in utterance scores is 0.36 points for Qwen2.5-Omni, 0.16 points for Gemini-3-Flash Preview, and 0.42 points for Qwen3-Omni-Thinking on the 1–5 scale. All paired 95% confidence intervals for the correlation changes include zero. These comparisons do not establish statistical equivalence between the extractors, but they suggest that the observed agreement is not highly sensitive to this particular MFCC implementation choice.
Conflicting tags. Table R3 examines how agreement with listeners changes under audio-only judging, correct acoustic evidence, and deliberately incorrect tags. In the incorrect-tag condition, we replace the global duration and word-count tags with valid categories that contradict the original measurements. The original recording, target text, quantitative measurements, and word- and segment-level evidence are held fixed between the correct- and incorrect-evidence conditions.For all 100 utterances, we collected three valid Qwen3-Omni-Thinking responses per utterance and condition. Table R3 reports the results from these runs, with 95% confidence intervals computed from 20,000 paired utterance-level bootstrap resamples.
Table R3. Effects of conflicting acoustic tags on Humanlike-100
| Conflicting-tag comparison | Audio-only r / ρ | Correct evidence r / ρ | Incorrect tags r / ρ | Δr, incorrect − correct [95% CI] | Δρ, incorrect − correct [95% CI] |
|---|---|---|---|---|---|
| Qwen3-Omni-30B-A3B-Thinking | 0.273 / 0.361 | 0.364 / 0.424 | 0.275 / 0.350 | −0.089 [−0.288, 0.119] | −0.074 [−0.274, 0.133] |
As shown in Table R3, the correct-evidence condition achieves the highest agreement with listeners on both metrics. With incorrect tags, the Pearson and Spearman correlations are close to the audio-only values, and the paired confidence intervals for the changes relative to the correct-evidence condition include zero. Taken together, these results suggest that correct structured evidence can improve agreement with listeners, whereas deliberately conflicting tags do not provide a comparable benefit and may largely reduce the evaluation to the audio-only condition.
Role of the audio input. In a separate ablation with the same backbone, evidence, and target text, withholding audio gave r / ρ = 0.238 / 0.231, compared with 0.364 / 0.424 when audio was supplied. Adding audio increased Spearman agreement by +0.194 [0.026, 0.367] and Pearson agreement by +0.127 [−0.046, 0.306]. The positive Spearman interval supports a contribution from listening in this configuration.
Together, these analyses suggest that HEAR is not dependent on one specific MFCC implementation and that the original audio provides useful information when structured evidence is incomplete or potentially inconsistent. The conflicting-tag analysis further indicates that correct acoustic evidence is more useful than deliberately contradictory evidence, while the AudioLLM’s judgment remains grounded in the original recording.
Thank you again for helping us improve the paper and hope our response can resolve your concerns! We would be very grateful if you would consider re-evaluating our work in light of these changes. Please let us know if you have any further questions.
Response Part 1/2: Emotional, Stylistic, and Textual Coverage (Weakness 1)
Dear Reviewer 2zXg,
We appreciate your thoughtful comments on the coverage of Humanlike-100 and the robustness of our acoustic evidence. We have provided a more detailed breakdown of emotions, speaking styles, and text types, along with results from four external benchmarks. We also report a comparison of two MFCC extractors, the test with conflicting acoustic tags, and the ablation that measures the effect of withholding the original audio. Our responses to your two concerns follow.
We thank the reviewer for requesting more detail on Humanlike-100’s emotional, stylistic, textual, and score coverage. We provide this breakdown below and then summarize results on four additional generated-speech tasks.
Dataset Composition and Coverage. With 100 utterances, we sought to capture variation in both what is spoken and how it is delivered. For the 50 synthetic utterances, we selected ten TTS systems with different expressive capabilities to include outputs ranging from highly human-like to more mechanical delivery. The input texts span conversational, narrative, and expository material, as well as phonetically challenging passages. For the 50 human recordings, we selected speech behaviors relevant to human-likeness judgments, including hesitations, slips of the tongue, self-corrections, and laughter.Table R1 summarizes the distributions separately for human and synthetic speech. Vocal-event categories are not mutually exclusive, so a single utterance may appear in multiple categories.
Table R1: Humanlike-100 composition and score coverage by speech origin
As shown in Table R1, Humanlike-100 contains 40 non-neutral utterances, and all four speaking styles are represented in both the human and TTS subsets. The TTS subset also covers all four transcript-content categories. Its listener-rated scores span a broad range of perceived human-likeness: 13 of the 50 TTS outputs received scores of at most 2, while 11 received scores above 4. These distributions provide coverage of varied emotional, stylistic, textual, and vocal conditions within the compact evaluation set.
Generalization to other generated-speech tasks. We evaluated HEAR on four additional datasets using the criteria of each task and the corresponding audio-only judge as the control.
Across all four datasets, HEAR achieved higher agreement with human judgments than the corresponding audio-only judge. These results suggest that the observed improvements extend beyond Humanlike-100 to both TTS naturalness evaluation and preference judgments for model-generated replies to spoken instructions.