acceptodds
Under review as a conference paper at ICLR 2027

Hearing Has a Computable Ceiling

Abstract

Spoken agents are built and evaluated on three assumptions: that the voice carries information the words do not; that a model's loss with and without audio measures what the audio adds; and that speech synthesized from a label can stand in for a person. We prove a ceiling on what hearing can add and test all three assumptions against it. Let be the information, in nats, that the voice carries about the speaker's hidden state beyond her words: no policy gains more than from hearing in one decision, and a turn-by-turn version bounds a whole conversation. Both bounds have sharp constants and are machine-checked in Lean 4. Because is a difference of two log-losses, a system can estimate it from its own conversations, given labels of the user's state and no transcript-reading judge. On television dialogue, the words already carry to percent of what the voice says about the speaker's emotion. The with-and-without comparison overstates what audio adds by the text model's shortfall, six-fold for a bag-of-words text model, and audio from the wrong utterance reproduces most of two off-the-shelf models' apparent gain. Speech synthesized from an emotion label, using instructions we wrote, retains little of what the speaker's own delivery carries. One off-the-shelf model collects a third of the ceiling on dialogue, the other nothing detectable.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spoken dialogue agents, audio language models, value of information, conditional mutual information, speech emotion, evaluation methodology, text-to-speech, formal verification

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_6TMpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recently there are several papers working on voice agents, and the advantage of voice agents over text-only agents is that it has access to voice information, thereby they are theoretically better at knowing the true user intent during a conversation. In this paper, the authors argue that actual payoff for a conversational agent from the additional voice cues are smaller than people think. They setup a mathematical formulation to derive an upper bound for payoff when the model has access to the voice other than just text. They empirically conduct the estimation on multiple benchmarks for predicting speaker's emotion, proving that most of the accuracy gain comes from text transcript only.

Critical strengths
  1. Nice impact. This paper motivated speech researchers to rethink how much speech can add to a conversation agent. I think this is a good contribution to the speech community.
  2. Meticulous math setup together with empirical verification. The authors show not only rigid mathematical proofs, they also show empirical results on real datasets.
Critical weaknesses
  1. Evaluation Task is biased. The authors choose to evaluate on emotion prediction tasks, which is quite biased. Emotion classification is one of the tasks that is easier to be predict from text transcript alone. Humans often choose different words according to their emotion. Hence, showing that the gain with speech included is quite predictable. I would suggest considering other tasks such that the transcripts are the same but with different speaking delivery, the intent is significantly different. E.g. stress detection. A same sentence can have different meanings if the speaker puts stress on different words.

  2. Evaluation results and framing is somehow misleading. Together with the payoff upper limit and the evaluation result (on emotion), it somehow lead the readers to the takeaway or conclusions that "voice adds far less than it seems to". However, think of a sarcastic speech. Based on the different ways of speaking the same sentence, the user’s intent can be either sarcastic or not sarcastic (which is literally the text transcript meaning). In this case, the payoff gain is 0.50.5 which is indeed smaller than the authors’ proposed upper limit (log⁡2\log{2}). However, for this task this means that without audio, the accuracy is 50% (random guessing), while with audio (assuming perfect prediction), it is 100%. This is a non-trivial difference. It’s a difference between a completely functional system and a completely useless system.

  3. Regarding the payoff as uniform. Even if the payoff of integrating speech is from 90% to 92%, this 2% might not be equally important as the original 90%. It is unknown how this 2% matters to the user or whether its a high-stakes scenario.

Questions for authors

Suggestions

  1. I think it might be better to add some visual and concrete examples to help the readers understand the formulas faster. E.g. You can use an example of prediction task and show the calculation of payoff and the bound.
  2. The framing as the "voice adds far less than it seems to" might be too arbitrary. Perhaps change the wording.
Reviewer_rew7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an interesting and important question: Given a transcript, how much additional task-relevant information does audio provide, and how much can that information improve decision-making? Through theoretical analysis and empirical experiments, the authors present an insightful framework for understanding the incremental value of the audio channel in spoken-language systems. However, the empirical analysis focuses primarily on emotional information in speech, with a relatively narrow range of data and prediction targets. The applicability of these findings to broader audio understanding tasks therefore remains unclear. The paper should distinguish more explicitly between the generality of its theoretical framework and the scope of its empirical findings.

Critical strengths

The experimental design is insightful. In particular, comparisons across text baselines of varying strength, together with shuffled-audio controls, demonstrate that performance improvements from adding audio do not necessarily indicate that a model effectively uses the additional information carried by the audio. Furthermore, by connecting information-theoretic analysis to decision-making utility, the paper provides a clear theoretical framework and a practical estimation procedure for quantifying the incremental information that an audio representation provides about a target state, conditional on the text.

Critical weaknesses
  • There is a gap between the scope of the empirical study and the breadth of its conclusions.

    Audio encompasses not only speech but also music and environmental sounds, for which substantial task-relevant information is absent from speech transcripts. Since the paper primarily examines emotion in speech, its experiments do not support broad conclusions about the informational value of audio in general. This does not mean that the authors must cover every audio task. Rather, the title, abstract, and conclusion should delimit the scope more clearly, so that readers do not interpret “limited incremental value in specific speech emotion tasks” as “limited incremental value of audio as a whole.”

  • The incremental value of audio is highly task-dependent, and emotion labels do not capture all important non-textual information.

    In conversational emotion recognition, audio may provide only modest additional benefits when the transcript and dialogue context already offer sufficient cues. However, this finding does not directly generalize to other tasks. For example, speaker-specific acoustic characteristics may be central to speaker identification, while background sounds may be essential to acoustic scene understanding. Even within emotion-related tasks, audio may play a different role when the words and tone of voice conflict or when textual cues are insufficient. I encourage the authors to evaluate such tasks or subsets, or at least discuss more thoroughly the extent to which the observed low incremental value reflects the chosen dataset and prediction target, rather than a limitation of the audio modality itself.

Questions for authors

To what extent does the reported 92–95% redundancy reflect the use of scripted dialogue and emotion labels, rather than a broader limitation on the incremental value of audio? Do your existing analyses show whether this finding holds for utterances with weak textual emotion cues or conflicting lexical and vocal cues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.