- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Recently there are several papers working on voice agents, and the advantage of voice agents over text-only agents is that it has access to voice information, thereby they are theoretically better at knowing the true user intent during a conversation. In this paper, the authors argue that actual payoff for a conversational agent from the additional voice cues are smaller than people think. They setup a mathematical formulation to derive an upper bound for payoff when the model has access to the voice other than just text. They empirically conduct the estimation on multiple benchmarks for predicting speaker's emotion, proving that most of the accuracy gain comes from text transcript only.
Critical strengths
- Nice impact. This paper motivated speech researchers to rethink how much speech can add to a conversation agent. I think this is a good contribution to the speech community.
- Meticulous math setup together with empirical verification. The authors show not only rigid mathematical proofs, they also show empirical results on real datasets.
Critical weaknesses
-
Evaluation Task is biased. The authors choose to evaluate on emotion prediction tasks, which is quite biased. Emotion classification is one of the tasks that is easier to be predict from text transcript alone. Humans often choose different words according to their emotion. Hence, showing that the gain with speech included is quite predictable. I would suggest considering other tasks such that the transcripts are the same but with different speaking delivery, the intent is significantly different. E.g. stress detection. A same sentence can have different meanings if the speaker puts stress on different words.
-
Evaluation results and framing is somehow misleading. Together with the payoff upper limit and the evaluation result (on emotion), it somehow lead the readers to the takeaway or conclusions that "voice adds far less than it seems to". However, think of a sarcastic speech. Based on the different ways of speaking the same sentence, the user’s intent can be either sarcastic or not sarcastic (which is literally the text transcript meaning). In this case, the payoff gain is which is indeed smaller than the authors’ proposed upper limit (). However, for this task this means that without audio, the accuracy is 50% (random guessing), while with audio (assuming perfect prediction), it is 100%. This is a non-trivial difference. It’s a difference between a completely functional system and a completely useless system.
-
Regarding the payoff as uniform. Even if the payoff of integrating speech is from 90% to 92%, this 2% might not be equally important as the original 90%. It is unknown how this 2% matters to the user or whether its a high-stakes scenario.
Questions for authors
Suggestions
- I think it might be better to add some visual and concrete examples to help the readers understand the formulas faster. E.g. You can use an example of prediction task and show the calculation of payoff and the bound.
- The framing as the "voice adds far less than it seems to" might be too arbitrary. Perhaps change the wording.