- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The authors introduced Voca, a benchmark that is supposed to evaluate voice companionship in audio language models. Their goal is to understand if models perceive users beyond what they say; focusing on how it is said. The benchmark contains examples across four dimensions: User State Understanding, Emotional Interaction, Proactive Care, and Safe Companion Behavior. The examples are composed of naturally occurring audio and also synthesized ones to better reflect their use cases. Fourteen systems including end-to-end, real-time, and cascaded architectures are evaluated and their main finding is that current models do not respond to implicit user needs conveyed through vocal cues in their voice. The authors also proposed VocaAgent, an inference-time multi-agent that decomposes acoustic and vocal information when it is present in a recorded speech. Overall i think the paper addressed an important and underexplored area that is vocal proactivity in voice systems. But I have concerns about the ecological validity of some of their tasks and the interpretation of results in their experiments.
Critical strengths
-
The problem their benchmark claims to evaluate is well scoped and conceptually relevant. Evaluating whether models can use vocal cues when explicitly asked and in the same vein use such cues when the user hasn’t verbalized thai preference is meaningful. The paper identifies a critical gap between foundational audio understanding and more complex interactional behavior.
-
I believe that the categories that they care about is quite broad, user-state understanding, emotional interaction, proactive care, and safety provides a more comprehensive view of spoken interaction than evaluating isolated perception tasks alone.
-
Very often when people release benchmarks, there is usually little to no thoughts on how to improve the benchmarks. Introducing Vocaagent as a potential non-verbal provide useful starting points for understanding why models may fail to act on vocal information available in their input and how to force them to act.
Critical weaknesses
-
The paper argues that proactive care involves acting on a cue appropriately rather than just perceiving them. However, I am not very convinced that the benchmark distinguishes failure to act on a vocal cue from the model making a reasonable decision not to infer a user’s preference based on how they sound. Vocal cues do not always uniquely map to an emotional state that requires a desired response. Crying, for instance, may suggest distress, but such a signal does not mean that the user wants emotional support. The examples in the first figure do not make a clear case that the intended response is uniquely justified by the available vocal evidence. Right now it seems like your evaluation will penalize a model that avoids making a strong assumption about a user's emotional state, thereby not acting on it. Are there examples where the correct behavior is to ask a clarifying question or acknowledge uncertainty rather than infer a particular emotional state? Are there negative-control examples where vocal cues are present but do not warrant an intervention?
-
The example in Figure 1 does not make a clear case that the intended response is uniquely justified by the available vocal evidence. As it stands, the evaluation seems to penalize a model that avoids making a strong assumption about a user’s emotional state. Also, it seems like every proactive care sample is a positive case during the data construction, so there are no negative controls where vocal cues are present but intervention is unwarranted. Are there examples where the correct behavior is to ask a clarifying question or acknowledge uncertainty rather than infer a particular emotional state?
-
The benchmark reuses data from existing benchmarks in combination with newly synthesized speech samples. While the authors describe automatic quality checks for the synthesized speech, they do not guarantee that the resulting audio sounds natural or that the intended cues are perceived reliably by human listeners. I would also like to point out the “emotion-transition” samples which are built by concatenating independent ESD utterances. Does this really preserve ecological validity of the tasks ? Utterances that individually convey the intended emotions may sound unnatural when stitched together, or the transition may be heard differently from what the label assumes. Did human annotators verify naturalness and that the cues are well perceived in any of the synthesized or constructed audios ?
-
The ground truth for the speaker identity safety task seems quite unclear. One speaker discloses personal information, a different speaker later asks for it, and the model is expected to withhold it. Why is withholding always the right behavior? If the audio input is self-contained, the model could reasonably assume both speakers are in the same conversation and the second speaker is just clarifying. “Private information” is also underspecified: does it mean PII or something explicitly marked confidential, or any personal detail? All of these categories might not warrant the same behavior, you know. In your task design, General safety and role/permission safety seem much more objective, which may explain why models do well on them.
-
The evaluation of emotional interaction and proactive care is cascaded and seems pretty loaded. GPT-5.6-sol generates sample-specific criteria, and Gemini 2.5 Flash judges responses against them. The human agreement study only checks whether the judge applies the criteria correctly, not whether the generated criteria themselves are right. A judge might apply a criteria that is not complete or based on a wrong assumption about the user’s needs. In Proactive Care, for example, a criterion could reward acknowledging a presumed emotional state that the audio doesn’t really support. In the paper, the authors seems to suggest that criteria may be generated using each model’s response. If so, are models being scored against different rubrics?
-
The proposed VocaAgent framework separates cue monitoring from response deliberation. The framework assumes that an auxiliary cue monitoring model can reliably identify relevant non-semantic cues and that the deliberative model can use those cues appropriately when generating a response. I would like to think that both assumptions require some sort of validation. If the monitor misidentifies an emotion or overstates what a cue implies, the deliberative component may simply act on an incorrect interpretation, leading to cascaded errors. Also, if the monitor identifies a cue correctly, the deliberative component still needs to decide whether it warrants action, how strongly to act, and whether other contextual information should override it. I am also not sure the gains come from separating monitoring and deliberation specifically. Do you have any ablations or analysis that shows that the monitoring models actually identify relevant cues beyond just the improvement in task scores.
Questions for authors
See weaknesses above
8 comments
Response to Reviewer ZzXV(2/5)
W4 Revising and Strengthening the Safe Companion Behavior Evaluation
Our Speaker Identity Safety samples concern requests for information that is explicitly protected by confidentiality restrictions. The second speaker requests disclosure beyond the permissions established by the original speaker, rather than merely adding context or clarifying an earlier request. The expected safety behavior therefore follows from the confidentiality and authorization conditions specified in the scenario. We revised and strengthened our self-constructed safety samples using GPT-5.6-sol to make these conditions clearer, while keeping the dataset size unchanged. The revisions clarify information ownership, confidentiality requirements, speaker identities, and authorization boundaries. We then synthesized the revised text into speech using TTS and checked the consistency between the text, audio, and intended safety behavior. The three subtasks address the following safety requirements:
- General Safety. We use requests from AdvBench, HarmBench, and JailbreakBench to evaluate whether models avoid facilitating harmful or malicious activities. These existing benchmark requests are retained rather than rewritten.
- Role and Permission Safety. We construct scenarios that require models to provide assistance within their roles, capabilities, and granted authority. For example, models should avoid presenting general health advice as a definitive diagnosis or making major personal decisions on behalf of users. Correct behavior requires respecting the relevant boundary while providing appropriate assistance; simply refusing every request is insufficient.
- Speaker Identity Safety. We construct scenarios involving two speakers with distinct voices. Speaker A provides information together with explicit disclosure restrictions, and Speaker B subsequently requests that protected information without the required authorization. The protected content includes passwords, access credentials, sensitive personal information, and other details explicitly restricted by the original speaker.
For example, when Speaker A provides an access credential and explicitly prohibits sharing it with others, a subsequent request from Speaker B to reveal that credential constitutes an unauthorized disclosure request. Appearing in the same conversation does not itself grant Speaker B permission to access the information. The model should withhold the protected content and may explain the restriction, request authorization, or offer assistance within the permitted scope. Our evaluation thus targets the ability to maintain confidentiality and respect authorization boundaries across speakers. It does not treat every speaker change as a reason for refusal or assume that all ordinary personal details must be withheld. The updated results for all three safety dimensions are now reported in Table 9 of the revised manuscript.
Response to Reviewer ZzXV(3/5)
W5 Human Evaluation of Generated Criterion Validity and Evaluation Consistency.
We would like to clarify that our evaluation pipeline consists of two separate stages: criterion generation and response evaluation. GPT-5.6-Sol generates sample-specific criteria solely based on the original input and task requirements, before examining any model responses. Each sample has one fixed set of criteria, which is shared by all evaluated models. Therefore, different models are not evaluated against different rubrics, and their responses do not influence criterion generation.
We acknowledge that our original human agreement study (89.9% agreement, Cohen's κ = 0.779) only examined whether the Judge Model correctly applied the generated criteria, rather than whether the criteria themselves were valid. To address this issue, we conducted an additional independent human audit involving two annotators.
Experimental Setup. We selected 100 samples through stratified sampling with sub-capability quotas, including 50 from Emotional Interaction and 50 from Proactive Care. These samples contained 468 original criteria (256 and 212, respectively) and 122 original WAV recordings. Both annotators independently examined the original audio, text, task instructions, and generated criteria, without access to model responses or Judge scores.
Evaluation Criteria. Each criterion was independently assessed along three dimensions: (1) Evidence Grounding, whether the criterion was supported by the original audio, text, or task instructions; (2) Evaluation Relevance, whether it directly assessed the intended capability without imposing irrelevant requirements; and (3) Assumption Validity, whether it avoided unsupported inferences about the user's emotions, physical state, intentions, or needs. Each dimension was scored from 0 to 2, where 2 indicates fully satisfactory, 1 indicates partially satisfactory with some uncertainty or minor issues, and 0 indicates a clear problem. We additionally assessed Completeness at the sample level (100 samples per annotator), examining whether the entire set of criteria covered the important expected behaviors. Completeness was not treated as an independent criterion-level observation.
The detailed results are shown below.
| Annotator | Evaluation Dimension | 2 (Fully Satisfactory) | 1 (Partially Satisfactory) | 0 (Problematic) |
|---|---|---|---|---|
| A | Evidence Grounding | 413 (88.25%) | 45 (9.62%) | 10 (2.14%) |
| A | Evaluation Relevance | 416 (88.89%) | 51 (10.90%) | 1 (0.21%) |
| A | Assumption Validity | 413 (88.25%) | 47 (10.04%) | 8 (1.71%) |
| A | Completeness | 88 (88.00%) | 12 (12.00%) | 0 (0.00%) |
| B | Evidence Grounding | 463 (98.93%) | 3 (0.64%) | 2 (0.43%) |
| B | Evaluation Relevance | 468 (100.00%) | 0 (0.00%) | 0 (0.00%) |
| B | Assumption Validity | 463 (98.93%) | 3 (0.64%) | 2 (0.43%) |
| B | Completeness | 97 (97.00%) | 2 (2.00%) | 1 (1.00%) |
Inter-Annotator Agreement. We further examined agreement between the two annotators at the criterion level. A criterion was classified as Valid only when it received a score of 2 on all three criterion-level dimensions; criteria classified as Partially Valid or Invalid were grouped as Not Valid. Completeness was excluded from this criterion-level classification because it assesses the entire set of criteria for each sample. The paired results were:
| Agreement | Number of Criteria | Percentage |
|---|---|---|
| Both classified as Valid | 413 | 88.25% |
| Both classified as Not Valid | 6 | 1.28% |
| Disagreement | 49 | 10.47% |
| Total | 468 | 100.00% |
The resulting binary inter-annotator agreement was 89.53% (419/468).
Overall, both annotators found the vast majority of generated criteria to be grounded in the original inputs, relevant to the intended evaluation tasks, and free from unsupported assumptions.
The revised manuscript now reports the criterion-validity audit in Appendix E.1 and the human–judge consistency study in Appendix E.2.
Response to Reviewer ZzXV(1/5)
Thank you for your valuable comments. We have carefully considered your suggestions, conducted additional experiments, and revised the manuscript accordingly. In response to the weaknesses you pointed out, our responses are as follows:
W1&W2 Clarifying the Scope of Proactive Care.
Voca focuses on a fundamental question: when users need care but do not explicitly express it, can models recognize the relevant non-semantic cues and respond appropriately? Our goal is not to determine whether every vocal cue warrants intervention, but to evaluate whether models can act when a care opportunity is clearly present. Our Paralinguistic Cue Care samples are single-turn interactions constructed as positive care opportunities, without prior conversational history. For example, when a user first asks for help while sounding noticeably exhausted, or when a nearby speeding vehicle and horn suggest a potential risk, a capable companion should at least acknowledge the cue or offer a gentle check-in while addressing the original request. This does not require making strong assumptions about the user's emotional state. Each sample has specific criteria for appropriate care, such as acknowledgement or a gentle check-in. Our results show that current models already struggle with these basic care opportunities, even without the additional demands of tracking prior interventions or resolving a complex conversational history. This motivates our focus on a foundational capability: recognizing a care-relevant cue and incorporating a proportionate response into the ongoing task. Reliable performance in these simpler settings is an important foundation for handling more complex interactions. The broader situations raised by the reviewer introduce further challenges, such as deciding whether care has already been provided or whether another intervention would be appropriate, which our positive-care evaluation is not designed to comprehensively assess. We additionally examine monitoring selectivity on 300 ordinary UltraChat utterances with no predefined care cues and premature intervention at checkpoints preceding the intended intervention conditions. These analyses are reported in Appendices F.2 and F.3. They complement the positive-care evaluation without establishing intervention calibration across unrestricted multi-turn interactions.
W3. Audio Quality Checks and the Scope of Concatenated Emotion Recognition
We would like to clarify that our audio construction involves two types of combination, both of which underwent quality checks.
The first is audio mixing, where speech is combined with background sounds, such as a user asking a question while an explosion is heard. Before data expansion, we manually listened to representative samples to ensure that the intended acoustic events were clearly audible and the resulting audio met our requirements. Only then did we proceed with further synthesis.
The second is sequential concatenation, used for Dynamic Emotion Understanding (Appendix G.2.1). We concatenate real, emotion-labeled ESD recordings with a silent interval between them, without modifying the original speech segments. The emotion labels are established before concatenation, and the original acoustic information is preserved. Models are asked to identify the emotion sequence from four options, including one correct answer and three distractors.
We believe that the naturalness of spontaneous emotional transitions should be distinguished from the validity of a controlled emotion-recognition task. Our goal is to evaluate whether models can recognize the emotions conveyed by the original recordings and their order, rather than whether the concatenated speech resembles a naturally occurring conversation. Although concatenation may not fully reproduce spontaneous emotional transitions, we believe this construction provides a reasonable and controlled setting for evaluating emotion understanding.