- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates prosody-dependent meaning transmission and recovery using five syntactically ambiguous English sentences with fixed lexical content. Models interpret human recordings in perception, while target-blind listeners interpret model-generated speech in production. Acoustic, behavioral, and rationale analyses, together with answer-order reversal, characterize model performance and response biases. The results motivate evaluating acoustic discriminability and naturalness separately from successful meaning transmission.
Critical strengths
- Holding lexical content fixed enables controlled evaluation of both perception and production. Target-blind human interpretation directly tests the communicative function of prosody.
- Acoustic discriminability, alignment with human timing patterns, naturalness, and meaning recovery are measured separately. This distinguishes detectable acoustic changes from changes that listeners can interpret meaningfully.
Critical weaknesses
- The evaluation uses only five English sentences. While these provide controlled contrasts, it is unclear whether they sufficiently support broader conclusions about model-level prosodic competence beyond these particular stimuli.
- Section 3.2 does not specify the actual production inputs, leaving unclear whether systems receive meaning descriptions, explicit boundary instructions, or example speech. These conditions test different combinations of interpretation and prosodic control. The knife prompt mismatch in Table 2 also requires clarification: does it affect only human–model alignment, or also the correspondence between generation targets and evaluation labels? The latter could affect meaning-recovery accuracy independently of model capability.
- A low pause–report correlation does not rule out reliance on duration or F0. Describing Grok’s correlation of 0.025 as “almost no acoustic grounding” therefore appears too broad. The diagnosis should be restricted to pause–report alignment, as for OpenAI in Table 4, or supported by evidence covering other cues. Narrowing the claim may resolve this concern without additional experiments, but the paper’s contribution should then be assessed within that narrower scope.
Questions for authors
- As a possible extension, how could this evaluation be adapted to other languages?
- Could the actual inputs for each production system be provided? In particular, does Grok STS receive speech already carrying the intended prosody?
- Which interpretations were mismatched for the knife sentence in Table 2, and how were this item and its labels handled in Figures 2–3? If the mismatch affected those results, do the conclusions hold after correction?
- Figure 3(b) appears to show approximately 68% human meaning recovery, whereas Section 4.2 reports 90%. Are both values correct under different evaluation conditions? Please clarify the recordings, listener groups, trial counts, and aggregation procedures underlying each result.
- How were boundary reports coded and validated? Do the grounding profiles refer specifically to pause–report alignment or to acoustic evidence more broadly?