acceptodds
Under review as a conference paper at ICLR 2027

HEAR: Human-likeness Evaluation with Acoustic Reasoning for Generative Speech

Abstract

Assessing the perceived human-likeness of generated speech is important but challenging. Listening tests are costly to scale, quality-oriented metrics do not directly measure this target, and raw-audio AudioLLM judges offer limited explicit acoustic evidence. We propose HEAR, a training-free method for Human-likeness Evaluation with Acoustic Reasoning. HEAR uses off-the-shelf tools and explicit rules to convert acoustic measurements into semantic tags organized at global, word, and segment levels. This representation links measured behavior to its location and surrounding speech. A dedicated human-likeness prompt guides an AudioLLM to judge the original audio using the target text and evidence, producing an overall human-likeness score, eight dimension-specific scores, and evidence-grounded reasons. On Humanlike-100, HEAR raises Qwen3-Omni-Thinking's Pearson and Spearman correlations with mean human overall ratings to and those of matched audio-only judging, respectively. HEAR improves overall agreement with human ratings across all five evaluated AudioLLM backbones. Ablation studies support the contributions of structured acoustic evidence and the human-likeness evaluation rubric. Beyond human-likeness evaluation, task-adapted HEAR improves audio-only judging on speech preference and perceptual-quality benchmarks. Code is available at https://github.com/HEAR-6/HEAR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-likeness evaluation, generative speech, AudioLLM-as-Judge, acoustic reasoning, text-to-speech

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2zXgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HEAR, a training-free framework that assesses the human-likeness of generated speech through acoustic reasoning. HEAR employs off-the-shelf tools to extract acoustic measurements and organizes them into structured semantic evidence at the global, word, and segment levels. An AudioLLM then evaluates the original audio together with the target text and acoustic evidence, yielding an overall human-likeness score, eight dimension-specific scores, and corresponding explanations. The authors also build Humanlike-100, a human-annotated evaluation dataset, and show stronger correlations with human judgments across five AudioLLM backbones. Further experiments on speech preference and perceptual-quality benchmarks confirm the effectiveness of the proposed framework.

Critical strengths

HEAR presents a training-free approach that integrates direct audio perception with structured acoustic evidence at the global, word, and segment levels. This design enables more fine-grained and interpretable human-likeness evaluation without the need for additional model training. It also strengthens correlation with human judgments across five AudioLLM backbones, with especially notable gains on Qwen3-Omni-Thinking. Ablation studies confirm the effectiveness of structured acoustic evidence, while further evaluations on speech preference and naturalness benchmarks support the broader applicability of the framework.

Critical weaknesses
  1. Humanlike-100 comprises only 100 speech samples, yet the paper offers no detailed analysis of how emotional expressions, speaking styles, or textual content are distributed. Such information is essential for judging whether the dataset adequately captures diverse speech scenarios and whether the reported improvements generalize to different types of generated speech.

  2. Although the paper reports tool-removal and additional-feature experiments, it lacks controlled comparisons that use alternative tools to measure the same acoustic properties. Moreover, the framework instructs the AudioLLM to prioritize the original audio when acoustic evidence conflicts with what it hears, but this behavior is not experimentally validated. Further experiments involving tool substitution and deliberately incorrect or conflicting acoustic tags would help establish the robustness and reliability of the proposed framework.

Questions for authors

Please refer to the Weakness above.

2 comments
AuthorsOpenReview ↗

Response Part 1/2: Emotional, Stylistic, and Textual Coverage (Weakness 1)

Dear Reviewer 2zXg,

We appreciate your thoughtful comments on the coverage of Humanlike-100 and the robustness of our acoustic evidence. We have provided a more detailed breakdown of emotions, speaking styles, and text types, along with results from four external benchmarks. We also report a comparison of two MFCC extractors, the test with conflicting acoustic tags, and the ablation that measures the effect of withholding the original audio. Our responses to your two concerns follow.


We thank the reviewer for requesting more detail on Humanlike-100’s emotional, stylistic, textual, and score coverage. We provide this breakdown below and then summarize results on four additional generated-speech tasks.

Dataset Composition and Coverage. With 100 utterances, we sought to capture variation in both what is spoken and how it is delivered. For the 50 synthetic utterances, we selected ten TTS systems with different expressive capabilities to include outputs ranging from highly human-like to more mechanical delivery. The input texts span conversational, narrative, and expository material, as well as phonetically challenging passages. For the 50 human recordings, we selected speech behaviors relevant to human-likeness judgments, including hesitations, slips of the tongue, self-corrections, and laughter.Table R1 summarizes the distributions separately for human and synthetic speech. Vocal-event categories are not mutually exclusive, so a single utterance may appear in multiple categories.

Table R1: Humanlike-100 composition and score coverage by speech origin

Attribute / categoryAll (n=100)Human (n=50)TTS (n=50)
Emotion
Neutral603030
Happy / amused19910
Sad220
Angry413
Frustrated1156
Surprised220
Disgusted110
Mixed emotion101
Speaking style
Conversational401921
Narrative1064
Formal / expository352114
Dramatic / expressive15411
Vocal events
Laughter1174
Audible breath753936
Hesitation16124
Repetition1697
Self-correction211
Transcript content
Everyday conversation / personal accounts522725
Literary / fictional narrative16610
Expository / factual / practical content281711
Phonetic challenges404
Transcript language
Chinese only492326
English only452520
Chinese and English words or names624
Duration
<5 s20911
5–<10 s402515
10–<20 s321517
≥20 s817
Listener-rated overall score
[1, 2]13013
(2, 3]15510
(3, 4]372116
(4, 5]352411

As shown in Table R1, Humanlike-100 contains 40 non-neutral utterances, and all four speaking styles are represented in both the human and TTS subsets. The TTS subset also covers all four transcript-content categories. Its listener-rated scores span a broad range of perceived human-likeness: 13 of the 50 TTS outputs received scores of at most 2, while 11 received scores above 4. These distributions provide coverage of varied emotional, stylistic, textual, and vocal conditions within the compact evaluation set.

Generalization to other generated-speech tasks. We evaluated HEAR on four additional datasets using the criteria of each task and the corresponding audio-only judge as the control.

  • SpeechJudge-Eval. The judge chooses which of two TTS renditions of the same text sounds more natural. The shared text lets the comparison focus on how the text is spoken.
  • TTSDS2. The judge scores the naturalness of TTS outputs in four domains spanning clean and noisy read speech, in-the-wild speech, and child speech. We compare system-average scores with human naturalness ratings to assess whether the judge captures differences across TTS systems.
  • SpeakBench. The judge compares two generated replies to a spoken instruction, which can request a particular vocal style or paralinguistic behavior. HEAR’s evidence helps assess how the replies sound and whether their delivery suits the instruction.
  • S2S-Arena. The judge compares generated replies in instruction-following, speech-perception, and conversational scenarios. As on SpeakBench, HEAR supplies acoustic evidence for voice-quality and paralinguistics judgments that inform the overall preference.

Across all four datasets, HEAR achieved higher agreement with human judgments than the corresponding audio-only judge. These results suggest that the observed improvements extend beyond Humanlike-100 to both TTS naturalness evaluation and preference judgments for model-generated replies to spoken instructions.

AuthorsOpenReview ↗

Response Part 2/2: Tool Substitution and Conflicting Evidence (Weakness 2)

Thank you for suggesting these controls. We examine whether HEAR is sensitive to the choice of MFCC extractor and how its scores respond to deliberately conflicting tags.

Tool substitution. We compared Librosa and Torchaudio/Kaldi configurations for extracting 13 MFCC coefficients. The configurations use different frame and Mel-filter settings. We aligned the scales of the resulting measurements using 60 separate, unlabeled clips. Audio, target text, prompt, and non-MFCC evidence were held fixed, and each utterance score averages three ratings. Table R2 shows how the swap changes Pearson and Spearman agreement with listeners for each judge.

Table R2. MFCC substitution on Humanlike-100

MFCC comparisonLibrosa r / ρTorchaudio/Kaldi r / ρΔr [95% CI]Δρ [95% CI]
Qwen2.5-Omni0.168 / 0.1370.253 / 0.282+0.084 [−0.091, 0.267]+0.144 [−0.031, 0.323]
Gemini-3-Flash Preview0.279 / 0.3300.319 / 0.356+0.040 [−0.026, 0.114]+0.026 [−0.045, 0.100]
Qwen3-Omni-Thinking0.299 / 0.3360.340 / 0.364+0.041 [−0.103, 0.200]+0.028 [−0.106, 0.168]

As shown in Table R2, replacing the Librosa extractor with the Torchaudio/Kaldi configuration produces small changes in the correlations. The mean absolute difference in utterance scores is 0.36 points for Qwen2.5-Omni, 0.16 points for Gemini-3-Flash Preview, and 0.42 points for Qwen3-Omni-Thinking on the 1–5 scale. All paired 95% confidence intervals for the correlation changes include zero. These comparisons do not establish statistical equivalence between the extractors, but they suggest that the observed agreement is not highly sensitive to this particular MFCC implementation choice.

Conflicting tags. Table R3 examines how agreement with listeners changes under audio-only judging, correct acoustic evidence, and deliberately incorrect tags. In the incorrect-tag condition, we replace the global duration and word-count tags with valid categories that contradict the original measurements. The original recording, target text, quantitative measurements, and word- and segment-level evidence are held fixed between the correct- and incorrect-evidence conditions.For all 100 utterances, we collected three valid Qwen3-Omni-Thinking responses per utterance and condition. Table R3 reports the results from these runs, with 95% confidence intervals computed from 20,000 paired utterance-level bootstrap resamples.

Table R3. Effects of conflicting acoustic tags on Humanlike-100

Conflicting-tag comparisonAudio-only r / ρCorrect evidence r / ρIncorrect tags r / ρΔr, incorrect − correct [95% CI]Δρ, incorrect − correct [95% CI]
Qwen3-Omni-30B-A3B-Thinking0.273 / 0.3610.364 / 0.4240.275 / 0.350−0.089 [−0.288, 0.119]−0.074 [−0.274, 0.133]

As shown in Table R3, the correct-evidence condition achieves the highest agreement with listeners on both metrics. With incorrect tags, the Pearson and Spearman correlations are close to the audio-only values, and the paired confidence intervals for the changes relative to the correct-evidence condition include zero. Taken together, these results suggest that correct structured evidence can improve agreement with listeners, whereas deliberately conflicting tags do not provide a comparable benefit and may largely reduce the evaluation to the audio-only condition.

Role of the audio input. In a separate ablation with the same backbone, evidence, and target text, withholding audio gave r / ρ = 0.238 / 0.231, compared with 0.364 / 0.424 when audio was supplied. Adding audio increased Spearman agreement by +0.194 [0.026, 0.367] and Pearson agreement by +0.127 [−0.046, 0.306]. The positive Spearman interval supports a contribution from listening in this configuration.

Together, these analyses suggest that HEAR is not dependent on one specific MFCC implementation and that the original audio provides useful information when structured evidence is incomplete or potentially inconsistent. The conflicting-tag analysis further indicates that correct acoustic evidence is more useful than deliberately contradictory evidence, while the AudioLLM’s judgment remains grounded in the original recording.


Thank you again for helping us improve the paper and hope our response can resolve your concerns! We would be very grateful if you would consider re-evaluating our work in light of these changes. Please let us know if you have any further questions.

Reviewer_MaajOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEAR, a training-free pipeline for evaluating the perceived human-likeness of generated speech using AudioLLM judges. Instead of prompting an AudioLLM with raw audio alone, HEAR first runs eight external speech analysis tools (Whisper-large-v3, MFA, wavelet prosody analysis, Librosa, Praat, Respiro-en, YAMNet, Sidon, and ECAPA-TDNN) and applies hardcoded numerical thresholds to convert the measurements into categorical tags at global, segment, and word levels. This structured JSON evidence is passed alongside the audio and target text to an off-the-shelf AudioLLM under an 8-dimension human-likeness rubric. On a newly collected 100-utterance dataset (Humanlike-100, consisting of 50 real recordings and 50 TTS clips across 10 systems), adding HEAR improves correlation with human ratings across five AudioLLM backbones (e.g., raising Qwen3-Omni-Thinking from r=0.197 to 0.477). The authors also show gains over audio-only prompting on SpeechJudge-Eval, TTSDS2, SpeakBench, and S2S-Arena.

Critical strengths
  • The motivation is sensible: current AudioLLMs struggle with fine-grained temporal localization and subtle acoustic tracking (like exact pause durations or gradual speaker timbre drift) when prompted only with raw audio. Supplying localized timestamps and prosodic tags as auxiliary context is a practical way to ground the model's reasoning.
  • The experimental evaluation covers five different AudioLLM backbones and four external benchmarks (SpeechJudge-Eval, TTSDS2, SpeakBench, and S2S-Arena), and the authors provide very thorough reproducibility documentation on tool versions, prompts, and threshold rules in the appendix.
  • The ablation studies in Table 2 systematically test the removal of individual tools, evidence granularity levels (word vs. segment vs. global), representation formats (structured JSON vs. raw JSON vs. natural language), and rubric dimensions.
Critical weaknesses
  • The main benchmark, Humanlike-100, is very small (only 100 utterances in total, with just 5 utterances per TTS system) and mixes 50 in-the-wild human recordings (from datasets like IEMOCAP, Emilia, and SeniorTalk) with 50 synthetic TTS utterances. Moreover, the annotation rubric (Appendix B.1 and B.2) explicitly caps "reading-like or studio-style delivery" at 3 out of 5, penalizes "unnaturally clean or dead sound" under Acoustic Realism, and states that "perfectly flawless reading should not get highest scores" under Error Correction. Because synthetic TTS outputs are typically clean and fluent while real recordings from wild corpora contain room acoustics, background noise, and spontaneous disfluencies, pooling the 50 real and 50 synthetic clips creates a strong confounding effect (which also explains why UTMOSv2 and DNSMOS have negative correlations of -0.34 in Table 1). Any acoustic tool feature (e.g., Praat HNR/RT60, YAMNet non-speech events, or MFCC variance) that simply separates clean studio audio from wild recording conditions will boost correlation on the pooled 100 clips without necessarily measuring which TTS system sounds more human-like. The paper never reports correlations computed purely within the 50 generated TTS utterances.
  • Because HEAR is training-free, all design choices—which 8 tools to include, which features to drop, and the dozens of exact numerical thresholds in Table 3 (Appendix C.1)—had to be determined manually. There is no mention of a separate validation set used to tune these choices. Looking at Table 6 in Appendix D.2, adding standard pitch extractors (pYIN or CREPE), eGeMAPS, or WavLM/HuBERT features to HEAR causes Pearson r on Humanlike-100 to drop sharply from 0.477 down to 0.303–0.372. This strong sensitivity on a 100-sample test set raises concern that the specific combination of tools and heuristic thresholds in HEAR may be over-tuned to Humanlike-100.
  • From a methodological standpoint, the contribution is quite incremental relative to prior evidence-grounded evaluators like TRACE (Chandra et al., 2026), which already extracts acoustic cues into structured text blueprints for LLM-based speech evaluation. While HEAR feeds the JSON tags together with raw audio into an AudioLLM and organizes tags at word/segment levels, the pipeline is essentially a collection of off-the-shelf extractors and hand-crafted thresholding rules.
  • Section 4.1 and Appendix C.4 introduce a 2,702-pair dataset called "HEAR-H data resource", claiming it as part of the experimental setup, yet HEAR-H is never actually used or evaluated anywhere in the experiments in Section 5.
  • The computational overhead per utterance is substantial: running Whisper-large-v3, MFA forced alignment, Sidon speech enhancement, ECAPA-TDNN, YAMNet, Respiro-en, Praat, and wavelet analysis, followed by averaging three valid Thinking responses (up to 8k–16k tokens each). Despite this heavy pipeline, the gains on spoken dialogue benchmarks like SpeakBench (+1.0%) and S2S-Arena (+1.9%) are quite small, and on SpeechJudge-Eval and TTSDS2, HEAR still trails behind dedicated lightweight evaluators like SpeechJudge-BTRM (68.2% vs. 72.8%) and TTSDS 2.1.1 (Figure 2).
Questions for authors
  1. On Humanlike-100, what are the utterance-level and system-level correlations (Pearson r and Spearman rho) when evaluated strictly on the 50 generated TTS utterances (excluding the 50 real human recordings), and separately within the 50 real recordings? Does HEAR still significantly outperform audio-only judging when separating real vs. synthetic recording environments is no longer part of the task?
  2. How were the specific eight tools and the exact numerical thresholds in Table 3 selected? Were they tuned directly on Humanlike-100, or on a separate held-out development set? Also, why does adding standard pitch information (pYIN or CREPE) in Table 6 degrade correlation so drastically (from 0.477 to 0.303–0.361)?
  3. Why is the HEAR-H dataset introduced in Section 4.1 if no experiments or results on it are reported in the paper?
  4. What is the average wall-clock latency per utterance for running the Stage-1 tool extraction pipeline (including MFA alignment and Sidon restoration) plus the 3-pass AudioLLM Thinking inference?
6 comments
AuthorsOpenReview ↗

Response Part 1/5: Recording Confounds and Within-Group Evaluation (Weakness 1, Question 1)

Dear Reviewer Maaj,

We sincerely thank you for your careful review and constructive questions. We address each point below.


We thank the reviewer for noting that pooled human–synthetic evaluation may conflate recording conditions with human-likeness and for requesting separate utterance- and system-level analyses of human and synthetic speech. We address dataset design, recording characteristics, and within-group results below.

Dataset design. With 100 utterances, we sought to cover a range of conversational human-likeness by varying both what is spoken and how it is delivered. For the 50 synthetic utterances, sampling five outputs per system allowed us to include ten TTS systems with different expressive capabilities. Collectively, these outputs range from highly human-like to more mechanical delivery. The input texts span conversational, narrative, and expository material, as well as phonetically challenging passages. For the 50 human recordings, we selected speech behaviors relevant to human-likeness judgments, including slips of the tongue, repetitions, hesitations, self-corrections, and laughter.

Given our annotation budget, we prioritized dense, fine-grained judgments. Ten listeners rated all 100 utterances on overall human-likeness and eight diagnostic dimensions, yielding 9,000 ratings. We assessed the reliability of each utterance’s mean overall score using the absolute-agreement intraclass correlation coefficient for the ten-listener mean. The resulting ICC(A,10) was 0.950, indicating that this average provides a reliable utterance-level reference for comparing HEAR with audio-only judging. Although compact, the dataset combines expressive diversity with dense human annotation and serves as a focused evaluation set.

Rubric and recording characteristics. Our evaluation aims to assess perceived human-likeness in a way that agrees with human listeners. Recording characteristics can contribute to this perception alongside prosody, rhythm, and vocal behavior. For example, Terblanche et al. [1] found that background noise and GSM transmission made lower-quality synthetic speech more likely to be judged human. This provides a perceptual motivation for including Acoustic Realism as one of eight evaluation perspectives.

Evaluation by speech origin. We agree that evaluating human and synthetic speech separately, together with analyses under more comparable recording conditions, strengthens the evaluation. To address Q1 directly, Table R1 reports Pearson and Spearman correlations within the synthetic and human subsets, at the TTS system level, and on the pooled set. Audio-only judging and HEAR use our main Qwen3-Omni-Thinking configuration; UTMOSv2 and DNSMOS are included for comparison.

Table R1. Agreement with listeners by speech origin on Humanlike-100

SubsetAudio-only r / ρHEAR r / ρUTMOSv2 r / ρDNSMOS r / ρ
Generated TTS utterances (n = 50)0.021 / 0.0790.313 / 0.3420.119 / 0.1600.079 / 0.014
Human utterances (n = 50)0.368 / 0.3760.624 / 0.633−0.438 / −0.441−0.301 / −0.338
TTS system means (n = 10)−0.099 / 0.0740.447 / 0.543−0.268 / −0.212−0.119 / −0.188
All utterances (n = 100)0.197 / 0.2530.477 / 0.524−0.290 / −0.342−0.247 / −0.328

HEAR achieves higher Pearson and Spearman correlations than audio-only judging in both the TTS and human subsets. It also performs better at the TTS system level, where scores are averaged over five utterances and their corresponding human ratings for each system, and exceeds UTMOSv2 and DNSMOS in both the TTS-only and system-level comparisons. Two-sided Williams tests show significant Pearson gains within TTS speech (p = 0.0396) and human speech (p = 0.0109). These within-group and system-level results support improvements beyond the pooled human–synthetic comparison.

Evaluation within read-speech domains. HEAR’s gains persist within both the Clean and Noisy read-speech domains of TTSDS2.On TTSDS2, both domains use read-speech reference material. Clean uses LibriTTS references selected by signal-to-noise ratio, and Noisy uses LibriVox references collected without that filter. Relative to audio-only judging, HEAR raises Pearson/Spearman from 0.510/0.401 to 0.616/0.501 in Clean and from 0.571/0.420 to 0.654/0.600 in Noisy. The higher within-domain correlations show that the gains are also observed when the comparison is restricted to read-speech naturalness.

References

[1] Terblanche, C., Harrison, P., & Gully, A. J. (2021, June). Human Spoofing Detection Performance on Degraded Speech. In Interspeech (pp. 1738-1742).

AuthorsOpenReview ↗

Response Part 2/5: Tool and Threshold Selection and Additional Pitch Information (Weakness 2, Question 2)

We thank the reviewer for raising the concern that the manually selected tools and thresholds may be over-tuned to Humanlike-100, and for asking how these choices were made and why the addition of pitch features reduces the correlation. We address the selection of the tools and thresholds first, followed by the effect of adding pitch information.

Tool and threshold selection. The eight tools span transcription and alignment, prosody, vocal events, acoustic conditions, and speaker stability. We manually specified the cutoffs in Table 3 to map measurements to interpretable tags. We did not fit these cutoffs to Humanlike-100 ratings or tune them on a separately rated development set.For transparency, Table R2 summarizes the cutoffs from Table 3, how HEAR applies them, and the evidence supporting either the cutoffs themselves or the underlying measurements. Table R2. HEAR threshold definitions and supporting evidence

MeasurementFixed cutoffs and use in HEAR (Table 3)Rationale and scope of the evidence
Speaking rateAligned token count divided by the utterance or segment duration, in tokens/s. <1 very slow, [1, 2) slow, [2, 3.5) medium, [3.5, 5) fast, ≥5 very fast.Words/min correlates with listener tempo ratings (r = 0.91) [2]. Switchboard averages 196 words/min (3.27 words/s), giving an English reference scale [3]. The exact aligned-token bins are HEAR’s fixed descriptions of pace.
Inter-word gap and segmentationGap tags in seconds use the same intervals as token duration below. Split when the following gap is ≥0.30 s or the boundary score is >0.80.0.30 s has a direct segmentation precedent. Biron et al. varied the minimum silence from 100 to 1,900 ms on 60 spontaneous-English recordings. Boundary precision peaked at 300 ms in Fig. 6, with the reported peak range at 300–400 ms [4]. The remaining gap tags are fixed descriptive bins.
Aligned token durationIn seconds, ≤0 none, (0, 0.10) short, [0.10, 0.30) medium, [0.30, 0.70) long, [0.70, 1.50) very long, ≥1.50 ultra long.Medial and final words in spontaneous English intonation units average 0.201 s and 0.356 s, respectively [4]. These measurements give a reference scale for token durations. HEAR shares the fixed duration bins with inter-word gaps so both cues use the same descriptive tags.
Repetition flagIdentical adjacent tokens, ignoring case, separated by a gap <0.10 s.A short gap restricts the flag to immediate repetitions. The literature motivates examining pauses and prosody when interpreting a repetition [5], and 0.10 s is HEAR’s fixed trigger.
Utterance or segment durationIn seconds, ≤0 none, (0, 2) short, [2, 5) medium, [5, 10) long, [10, 20) very long, ≥20 ultra long.HEAR uses these fixed bins to summarize utterance or segment length.
Token count≤0 none, 1–3 few, 4–10 some, 11–30 many, >30 very many.HEAR uses these fixed bins to summarize how much text the span contains.
Prominence strength≤0.30 very weak, (0.30, 0.60] weak, (0.60, 0.90] medium, (0.90, 1.20] strong, >1.20 very strong.Wavelet analysis provides a continuous word-level prominence score [6]. HEAR’s fixed ordinal bins describe the strength of that cue.
Boundary strength≤0.10 very weak, (0.10, 0.40] weak, (0.40, 0.80] medium, (0.80, 1.20] strong, >1.20 very strong.Wavelet scores capture prosodic boundaries, including boundaries with little silence [6, 7]. HEAR splits at the start of its strong range (>0.80), a fixed score cutoff.

References

[2] Iwarsson, J., Naes, J., & Hollen, R. (2023). Measuring speaking rate: how do objective measurements correlate with audio-perceptual ratings?. Logopedics Phoniatrics Vocology, 48(2), 57-66.

[3] Yuan, J., Liberman, M., & Cieri, C. (2006). Towards an integrated understanding of speaking rate in conversation. In Proc. Interspeech 2006 (pp. paper-1795).

[4] Biron, T., Baum, D., Freche, D., Matalon, N., Ehrmann, N., Weinreb, E., ... & Moses, E. (2021). Automatic detection of prosodic boundaries in spontaneous speech. PloS one, 16(5), e0250969.

[5] Bartkova, K. (2005). Prosodic cues of spontaneous speech in French. In Proc. DiSS 2005 (pp. 21-25).

[6] Suni, A., Šimko, J., Aalto, D., & Vainio, M. (2017). Hierarchical representation and estimation of prosody using continuous wavelet transform. Computer Speech & Language, 45, 123-136.

[7] Inbar, M., Genzer, S., Perry, A., Grossman, E., & Landau, A. N. (2023). Intonation units in spontaneous speech evoke a neural response. The Journal of Neuroscience, 43(48), 8189-8200.

AuthorsOpenReview ↗

Response Part 2/5: Tool and Threshold Selection and Additional Pitch Information (Weakness 2, Question 2) — Continued

Table R2. HEAR threshold definitions and supporting evidence — Continued

MeasurementFixed cutoffs and use in HEAR (Table 3)Rationale and scope of the evidence
Prominent-token ratio<0.05 none, [0.05, 0.20) local, [0.20, 0.50) scattered, ≥0.50 dominant.These fixed reporting bins summarize how widely prominence is distributed.
Detected breath-interval ratioSummed detected breath duration divided by utterance duration. <0.10 stable speech dominant, [0.10, 0.25) speech dominant, [0.25, 0.40) speech weakened, ≥0.40 transitional weakened.Respiro-en supports detecting breath intervals [8]. HEAR’s fixed ratio bins summarize their coverage of the utterance.
Harmonics-to-noise ratio (HNR)In dB, <5 low/noisy, [5, 10) moderate, [10, 20) normal, ≥20 high/clear.Praat defines HNR as a measure of acoustic periodicity. HEAR uses fixed bins to summarize measured harmonicity.
Estimated reverberation time (RT60)In seconds, <0.05 abnormally short, [0.05, 0.10) very short, [0.10, 2) normal, [2, 5) long, ≥5 abnormally long. Missing estimates are marked not assessable.The room-acoustics standard defines the reverberation-time measurement. HEAR uses fixed bins to describe available estimates by decay length.
Mean MFCC standard deviation<1 abnormally stable/possible synthetic, [1, 5) stable, [5, 15) normal, ≥15 highly variable.The literature supports MFCCs as short-term spectral descriptors [9]. HEAR uses fixed bins on the extractor’s scale to summarize temporal variability.
Adjacent ECAPA cosine similarity<0.70 possible timbre drift, [0.70, 0.80) unstable, [0.80, 0.85) borderline, ≥0.85 stable. Windows are 6 s with a 3 s hop.Prior work supports comparing speaker embeddings and using overlapping windows [10, 11]. HEAR uses fixed similarity bins to flag possible voice changes between adjacent windows.

As shown in Table R2, the thresholds were not optimized against Humanlike-100 ratings. Where external precedents are available, they provide reference ranges for the corresponding measurements. The remaining thresholds are fixed descriptive bins on the relevant extractor scales. Thus, HEAR uses manually specified and interpretable rules rather than thresholds tuned to the evaluation set.

Additional pitch information. Table 6 adds utterance-level F0 statistics, such as the mean and standard deviation, to a judge that already hears the pitch contour in the original audio and receives word-aligned cues for emphasis and phrasing. Two utterances can have the same mean and standard deviation even when a pitch rise falls on different words and changes how the emphasis is perceived. The added statistics discard this timing information. On Humanlike-100, the added mean and standard-deviation fields also show weak correlations with the overall listener ratings (absolute Pearson r ≤ 0.142).Adding these summaries reduces the Pearson agreement with listeners from 0.477 to 0.361 with pYIN and 0.303 with CREPE. Agreement also decreases within the 50 TTS outputs alone, from 0.313 to 0.206 with pYIN and 0.084 with CREPE.

Overall, these results suggest that, in this configuration, retaining the original audio and word-aligned prosodic cues is preferable to adding standalone pitch summaries. The additional pitch statistics do not provide complementary information sufficient to offset the loss of timing and alignment information.

References

[8] Yang, D., Koriyama, T., & Saito, Y. (2024). Frame-wise breath detection with self-training: An exploration of enhancing breath naturalness in text-to-speech. arXiv preprint arXiv:2402.00288.

[9] Davis, S., & Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing, 28(4), 357-366.

[10] Desplanques, B., Thienpondt, J., & Demuynck, K. (2020). Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.07143.

[11] Dawalatabad, N., Ravanelli, M., Grondin, F., Thienpondt, J., Desplanques, B., & Na, H. (2021). ECAPA-TDNN embeddings for speaker diarization. arXiv preprint arXiv:2104.01466.

Reviewer_1mPoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HEAR, a training-free framework for evaluating the perceived human-likeness of generated speech. HEAR extracts acoustic measurements using existing tools, converts them into structured evidence at the utterance, word, and segment levels, and provides this evidence alongside the audio to an AudioLLM. The evaluator produces an overall human-likeness score, scores across eight perceptual dimensions, and explanations grounded in acoustic evidence.

Critical strengths
  • The paper introduces a training-free method, that adds acoustic measurements into Audio-LLM based evaluation.
  • Evaluations across 5 systems demonstrates that HEAR improves Pearson and Spearman correlation.
Critical weaknesses
  • The main evaluation uses only 100 utterances, with just five samples per TTS system. Without statistical significance testing, it is unclear how reliable the reported improvements are.
  • The evaluation rubric explicitly penalizes clean studio-style speech, raising the possibility that HEAR captures recording characteristics rather than human-likeness itself. Evaluating real and synthetic speech separately, ideally under matched recording conditions, would strengthen the conclusions.
Questions for authors
  • Can the authors report confidence intervals and paired significance tests for the improvements in Table 1 to establish whether the results are statistically significant?
  • Can the authors report the main correlations separately for real and synthetic speech? Does the improvement persist when controlling for recording conditions and speaking styles?
2 comments
AuthorsOpenReview ↗

Response Part 1/2 :Dataset Size and Statistical Significance (Weakness 1, Question 1)

Dear Reviewer 1mPo,

We sincerely appreciate your careful review and constructive feedback on statistical reliability and recording conditions. In response, we have clarified how Humanlike-100 was designed and annotated, as well as how recording conditions can affect perceived human-likeness. We also report confidence intervals and paired significance tests, separate results for human and synthetic speech, and results for the Clean and Noisy domains of TTSDS2. The detailed responses are listed below.


Thank you for raising this important concern regarding the dataset size and the reliability of the reported improvements.

Dataset design. With 100 utterances, we aimed to cover a range of conversational human-likeness by varying both the content and the delivery. For the 50 synthetic utterances, sampling five outputs from each system allowed us to include ten TTS systems with different expressive capabilities. Collectively, these outputs cover a range from highly human-like delivery to more mechanical delivery. The input texts span conversational, narrative, and expository material, as well as phonetically challenging passages. For the 50 human recordings, we selected recordings exhibiting speech behaviors relevant to human-likeness judgments, including slips of the tongue, repetitions, hesitations, self-corrections, and laughter.

Within our annotation budget, we prioritized dense, fine-grained judgments. Ten listeners rated each of the 100 utterances on overall human-likeness and eight diagnostic dimensions, yielding 9,000 ratings. To assess the reliability of the mean overall score for each utterance, we computed the absolute-agreement intraclass correlation coefficient for the mean rating of the ten listeners. The resulting ICC(A,10) was 0.950, indicating that the ten-listener average provides a reliable utterance-level reference for comparing HEAR with audio-only judging. Although compact, the resulting set combines expressive diversity with dense human annotation and serves its intended role as a focused utterance-level evaluation dataset.

Confidence intervals and paired significance tests. To directly address Question 1, Table R1 compares HEAR with audio-only judging on Humanlike-100 for the five backbones reported in Table 1. The table reports changes in utterance-level Pearson and Spearman correlations with human ratings, together with 95% paired bootstrap confidence intervals based on 20,000 resamples of utterances. We tested the Pearson correlation gains using two-sided Williams tests for dependent correlations, with Holm correction across the five backbones.

Table R1: Paired utterance-level correlation gains on Humanlike-100

BackboneΔPearson r [95% CI]ΔSpearman ρ [95% CI]Holm-adjusted p (Pearson)
Qwen2.5-Omni0.133 [-0.115, 0.374]0.160 [-0.060, 0.378]0.760
Qwen3-Omni-Instruct0.081 [-0.078, 0.243]0.058 [-0.114, 0.234]0.760
Qwen3-Omni-Thinking0.280 [0.091, 0.459]0.271 [0.090, 0.455]0.0044*
Gemini-3-Flash0.186 [0.071, 0.295]0.161 [0.036, 0.283]0.0280*
Gemini-3.5-Flash0.037 [-0.121, 0.199]0.036 [-0.094, 0.164]0.760

*p < 0.05 after Holm correction.

As shown in Table R1, HEAR significantly improves the Pearson correlation with human ratings under our main configuration, Qwen3-Omni-Thinking (Δr = 0.280, adjusted p = 0.0044), and for Gemini-3-Flash (Δr = 0.186, adjusted p = 0.0280). For these two backbones, the 95% confidence intervals for both the Pearson and Spearman gains lie entirely above zero, and the Pearson gains are significant after Holm correction. The remaining three backbones show positive estimated gains, although their confidence intervals include zero.

Additional external evaluations. o examine whether the observed improvements extend beyond Humanlike-100, we also compared HEAR with matched audio-only judges on four external datasets, using the evaluation criteria defined for each dataset. For TTS naturalness, same-text preference accuracy increased from 62.1% to 68.2% on SpeechJudge-Eval, while system-level Pearson/Spearman correlations increased from 0.660/0.671 to 0.806/0.711 on TTSDS2. For generated replies to spoken instructions, four-way preference accuracy increased from 59.4% to 60.4% on SpeakBench and from 45.9% to 47.8% on S2S-Arena. These results provide additional evidence that the gains are not limited to Humanlike-100 and extend across the evaluated generated-speech tasks.

AuthorsOpenReview ↗

Response Part 2/2: Recording Conditions and Speaking Styles (Weakness 2, Question 2)

Thank you for asking how recording conditions and speaking styles affect human-likeness judgments and whether HEAR's gains hold separately for human and synthetic speech.

Rubric and recording characteristics. Our evaluation is intended to assess perceived human-likeness insofar as it agrees with human listeners. Recording characteristics may contribute to this perception alongside prosody, rhythm, and vocal behavior. For example, Terblanche et al. [1] found that background noise and GSM transmission made lower-quality synthetic speech more likely to be judged as human. This provides a perceptual motivation for including Acoustic Realism as one of the eight evaluation perspectives.

Within-Group Evaluation by Speech Origin. We agree that evaluating human and synthetic speech separately, as well as examining more comparable recording conditions, provides a useful additional analysis. Table R2 reports Pearson and Spearman correlations separately for TTS and human speech, together with TTS system-level and pooled results. Audio-only judging and HEAR use our main Qwen3-Omni-Thinking configuration; UTMOSv2 and DNSMOS are included as reference systems.

Table R2: Agreement with listeners by speech origin on Humanlike-100

SubsetAudio-only r / ρHEAR r / ρUTMOSv2 r / ρDNSMOS r / ρ
Generated TTS utterances (n = 50)0.021 / 0.0790.313 / 0.3420.119 / 0.1600.079 / 0.014
Human utterances (n = 50)0.368 / 0.3760.624 / 0.633−0.438 / −0.441−0.301 / −0.338
TTS system means (n = 10)−0.099 / 0.0740.447 / 0.543−0.268 / −0.212−0.119 / −0.188
All utterances (n = 100)0.197 / 0.2530.477 / 0.524−0.290 / −0.342−0.247 / −0.328

As shown in Table R2, HEAR achieves higher correlations with listener ratings than audio-only judging within both the generated TTS subset and the human-speech subset. The improvement is also observed at the TTS system level and across all utterances. These within-group results provide evidence that the observed gains are not solely driven by the pooled distinction between human and synthetic speech.

Analysis within read-speech domains. To further examine whether the improvement persists under more comparable speaking conditions, we evaluated HEAR separately on the Clean and Noisy read-speech domains of TTSDS2. Both domains use read-speech reference material. The Clean domain uses LibriTTS references selected by signal-to-noise ratio, whereas the Noisy domain uses LibriVox references collected without this filter.

Relative to audio-only judging, HEAR increases the Pearson/Spearman correlations from 0.510/0.401 to 0.616/0.501 in the Clean domain, and from 0.571/0.420 to 0.654/0.600 in the Noisy domain. These within-domain improvements indicate that HEAR’s gains persist when the comparison is restricted to read-speech naturalness and are therefore not limited to the pooled comparison of speech with different origins or recording conditions.


Thank you again for your reviewing efforts. We’re happy to address any more comments or concerns you may have.

References

[1] Terblanche, C., Harrison, P., & Gully, A. J. (2021, June). Human Spoofing Detection Performance on Degraded Speech. In Interspeech (pp. 1738-1742).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear AC and SAC,

We are grateful for the opportunity to submit our work to ICLR 2027 and to respond to the reviewers’ feedback. We have carefully considered their comments and provided detailed clarifications, supplementary experiments, and statistical analyses in response.

Our detailed rebuttal addressed the concerns about the size and coverage of Humanlike-100, statistical significance, recording conditions and within-group evaluation, tool and threshold selection, the effect of additional pitch information, the methodological contribution relative to TRACE, the purpose and human evaluation of HEAR-H, computational cost and practical benefits, generalization across speech evaluation tasks, tool substitution, and robustness to conflicting acoustic evidence.

While we deeply value the review process and the opportunity to engage with reviewers, we are concerned about the lack of follow-up responses after we posted our detailed rebuttals and supplementary results. This absence of further discussion has limited our ability to address any remaining concerns and leaves us uncertain whether our clarifications and additional evidence were adequately considered.

We respectfully ask that the AC and SAC take this into account when evaluating the reviews and making a final decision. We hope that the original concerns will be reassessed in light of the explanations and experimental evidence provided during the rebuttal period, and that our responses will receive appropriate consideration alongside the original reviews to ensure a fair and balanced outcome.

We remain available to provide any further clarification or information as needed. Thank you again for your time, effort, and dedication to the review process.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.