acceptodds
Under review as a conference paper at ICLR 2027

Can Audio-Capable Language Models Predict Human Emotion Judgments of Infant Vocalizations? Raw Audio Alone Is Not Enough

Abstract

Audio-capable large language models can process raw audio, but whether they reproduce fine-grained human emotion judgments for non-linguistic vocalizations is unclear. We compared four strategies on 178 matched transformed infant vocalizations rated for happiness, sadness, anger, fear, and disgust: direct GPT-Audio prediction from raw audio, supervised prediction from engineered acoustic features, and two feature-based in-context learning conditions using acoustically similar labeled examples with GPT-Audio and GPT-4o-mini. Direct GPT-Audio showed little correspondence with human ratings (r = -0.173 to 0.016), whereas supervised feature-based models reached r = 0.883. Feature-based in-context learning produced positive correspondence for both GPT-Audio (r = 0.450 to 0.725) and GPT-4o-mini (r = 0.573 to 0.786), although both remained below the strongest supervised models. These findings suggest that structured acoustic representations and labeled demonstrations support model prediction of human affective judgments, while conventional supervised feature-based approaches achieved higher correlations in this study.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
affective computing, audio emotion recognition, acoustic features, large language models, in-context learning, human emotion perception, GPT-Audio

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HdWcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether audio-capable language models can predict human emotion judgments of infant vocalizations. The authors compare direct GPT-Audio predictions from raw audio, conventional supervised regressors using engineered acoustic features, and feature-based in-context learning with GPT-Audio and GPT-4o-mini. Experiments use 178 synthetic stimuli derived from ten original infant vocalizations, with human ratings across five emotion dimensions. The results show that supervised feature-based methods achieve the highest correlations, while feature-based in-context learning outperforms direct raw-audio prompting.

Critical strengths

None

Critical weaknesses

1. Limited dataset validity and unreliable generalization: The evaluation uses only 178 synthetic stimuli derived from ten original vocalizations, raising serious concerns about dataset diversity and source-specific artifacts. Stimulus-level cross-validation allows transformed recordings sharing the same original sources to appear in both training and test sets, potentially inflating performance without demonstrating generalization to unseen infant vocalizations. Furthermore, the paper does not adequately characterize the source infants or report inter-annotator reliability or a human-rating noise ceiling. Given the small number of independent source recordings and the absence of source-disjoint evaluation, the reported correlations provide insufficient evidence of robust, generalizable emotion prediction.

2. Limited novelty, insufficient motivation, and inadequate experimental comparisons: Although the exact GPT-Audio comparison is new, Cooing, Crying, and Babbling: A Link between Music and Prelinguistic Communication (Byrd et al., 2012) previously investigated acoustic correlates of human emotion judgments using the same dataset, while Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance (Chen et al., 2026) already examined acoustic emotion cues in audio-language models. The paper does not introduce a new methodology or demonstrate a sufficiently distinct insight beyond this prior literature, and its central comparison is not controlled: raw-audio and feature-based conditions differ in labeled supervision, output format, and sampling procedures. The evaluation is also restricted to two GPT-based models, with only GPT-Audio tested on raw waveforms, preventing broader conclusions about audio-capable language models. Consequently, the experiments neither isolate why feature-based prompting performs better nor establish a substantial new methodological or generalizable empirical finding. These limitations significantly weaken the paper's motivation, technical contribution, and evidence for its central conclusions.

Questions for authors

Q1. Dataset independence and annotation reliability: How do the authors justify stimulus-level cross-validation when the 178 synthetic stimuli originate from only ten recordings? Could the authors provide source-disjoint evaluation, clarify the number and ages of the original infants, and report inter-annotator reliability or a human-rating noise ceiling?

Q2. Experimental controls: Why were raw-audio and feature-based conditions evaluated with different levels of labeled supervision, output formats, and sampling procedures? Could the authors provide matched comparisons, particularly raw-audio in-context learning with the same number of labeled demonstrations, to establish whether acoustic feature representation is responsible for the observed improvement?

Q3. Novelty and model coverage: What substantial new insight does this study establish beyond Byrd et al. (2012) and Chen et al. (2026)? Given the broader framing around audio-capable language models, why were additional models not evaluated under comparable raw-audio and feature-based conditions?

Reviewer_FXcSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines whether audio-capable LLMs can predict human emotion ratings of synthetic infant vocalizations. The authors compare zero-shot rating from audio by GPT-Audio against supervised models trained on engineered acoustic features and against in-context learning, in which LLMs receive these features as text with labeled examples. They find that direct audio prediction barely correlates with human ratings, feature-based in-context learning performs moderately, and supervised models perform best. They conclude that structured acoustic representations and labeled demonstrations are needed for audio LLMs to approximate human affective judgments.

Critical strengths

The paper raises an interesting and well-motivated question: whether audio-capable LMs can reproduce human affective judgments of non-linguistic vocalizations, where lexical cues are absent.

Critical weaknesses

W1. The main claim is not supported. The paper compares two settings for audio-capable language models. In the direct setting, the model takes each audio clip and rates it with no examples. In the feature-ICL setting, the model receives the clip’s numerical acoustic features as text, together with 25 acoustically similar samples and their human ratings. The second setting therefore exposes the model to the ratings of similar stimuli, which is highly informative on its own. As a result, the performance gap cannot be attributed to the input representation, and the title’s claim is not established. Moreover, it is unclear whether the LLM meaningfully uses the numerical features at all. Key ablations should be added, including (1) audio ICL, where audio clips paired with their ratings serve as demonstrations, and (2) zero-shot prediction from numerical features alone.

W2. Flawed framing and limited scope. The paper treats the comparison as “raw audio vs. engineered features.” However, audio LLMs do not operate on raw waveforms. They typically encode spectral features with learned audio encoders (e.g., the Whisper-based encoders in SALMONN and Qwen-Audio). The more relevant comparison is engineered vs. learned representations, yet no SSL or learned features are evaluated (e.g., WavLM, emotion2vec, Whisper encoder, CLAP). Moreover, only one closed-source audio LLM is tested (GPT-4o-mini is text-only), and no open-source audio LLMs are included. Conclusions about audio LLMs as a class are therefore not warranted.

W3. Dataset quality and leakage. All 180 stimuli derive from only 10 infant recordings collected from heterogeneous sources (a handheld recorder, a sound-effects website, and YouTube). Vocalization type, speaker, and recording condition are fully confounded. Each stimulus was created by mixing the spectral components of two recordings and manually adjusting pitch and amplitude to sound happy, sad, angry, or fearful. This yields four near-duplicate variants per recording pair.

Questions for authors

See weaknesses.

Reviewer_kcUPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper examines whether audio-capable LLMs can reproduce mean human emotion ratings for 178 transformed infant vocalizations, comparing zero-shot GPT-Audio prediction from raw audio, supervised regressors on engineered acoustic features, and feature-based in-context learning with GPT-Audio and GPT-4o-mini.

Critical strengths

Strength:

  • The paper asks whether general-purpose audio LLMs capture acoustic affective cues when lexical content is absent, and the negative result for zero-shot raw-audio prediction is a useful data point.
  • The authors are transparent about the limitations of their experimental design.
Critical weaknesses

Weakness:

  • The motivation needs stronger justification. The paper does not sufficiently explain why analysing non-linguistic vocalizations through general-purpose audio LLMs is meaningful, or why this is an important question for the community.
  • The novelty is limited. The paper compares existing models and standard strategies on an existing dataset, without proposing a new method or offering deeper analysis of why the models succeed or fail. The main findings are also largely expected.
  • The dataset is very small (178 used), which limits the validity of the findings. More importantly, source-level leakage undermines all cross-validated results. The stimuli are built from only 10 source recordings combined in 45 pairs, with four variants per pair. The Limitations section mentions this only briefly.
  • Inter-rater agreement should be reported to validate annotation quality.
  • Key baselines are missing. The paper evaluates a single audio LLM, with no open audio LLMs (e.g., Qwen2-Audio, SALMONN, Audio Flamingo) and, more importantly, no learned audio representations with a regression head (e.g., emotion2vec, wav2vec 2.0/HuBERT, CLAP embeddings), even the Related Work cites emotion2vec as a modern approach.
  • Page 7 contains a large blank space, which should be fixed.
  • The scope and fit for ICLR are limited. The contribution is an empirical comparison on one small synthetic dataset using one closed model, and it would need much broader evaluation to support general claims about audio LLMs. In its current form, the work seems better suited to a speech or affective computing venue such as Interspeech or ACII.
Questions for authors

Please see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.