acceptodds
Under review as a conference paper at ICLR 2027

Can You Hear My Mind? Catching Theory of Mind in Voice and Pragmatics

Abstract

The same words can convey different intentions depending on how they are spoken. Interpreting these cues alongside verbal content draws on Theory of Mind (ToM), the capacity to perceive and infer others’ mental states. Yet existing ToM benchmarks primarily assess mental-state reasoning from textual or visual events, while audio reserach largely focuses on state recognition rather than cognition. This leaves unclear how vocal delivery and pragmatics shape mind. We introduce Overtone, the first multimodal ToM benchmark to systematically assess how models use vocal and pragmatic cues to infer mental states. Overtone comprises 2,335 social-interaction scenarios with human-recorded speech, enables comprehensive, fine-grained evaluation across diverse classical ToM tasks and a broad spectrum of ToM abilities. A human study and evaluations of 16 speech and omni-modal models reveal that models struggle to maintain coherent mental-state representations and integrate vocal cues into ToM reasoning. Hence, we introduce PEAR (Prediction-Error-Adaptive Reasoning), a training-free ToM reasoning framework inspired by predictive coding in the human brain. Mental-state hypotheses guide the interpretation of multimodal cues, while metacognitive regulation the reasoning process. Extensive experiments show that PEAR improves performance across backbones, outperforms several reasoning baselines, and generalizes across diverse ToM datasets. Our findings highlight current multimodal models' limitations in inferring mental states from vocal and pragmatic cues. Together, our work provides a benchmark and reasoning framework for assessing and improving this capability, advancing multimodal social reasoning.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Theory of Mind, Multimodal Reasoning, Voice and Pragmatics, Cognitive-inspired Reasoning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_rMk9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Overtone, the first Theory of Mind (ToM) benchmark based on vocal and pragmatic cues. It also proposes PEAR (Prediction-Error-Adaptive Reasoning), a training-free ToM reasoning method. Evaluation on Overtone shows current models' limitations in inferring mental states from vocal and pragmatic cues. Experiments also demonstrate the efficacy of PEAR.

Critical strengths
  1. This paper studies an interesting but underexplored problem: ToM in voice and pragmatics. The proposed Overtone is the first benchmark on this topic. Most importantly, the benchmark evaluates how vocal expression can alter the social meaning of identical words, which cannot be explored by a text-only dataset. Experiments in Figure 4 present such an analysis.

  2. The experiments are comprehensive. They compare multiple types of baselines, apply the proposed PEAR to multiple backbone models, and conduct extensive analysis, particularly on audio gains.

Critical weaknesses
  1. The novelty of the proposed PEAR is not significant, as it is more engineering-oriented.

  2. The proposed Overtone lacks a shortcut audition. [r1] raises the point that many existing ToM benchmarks suffer from the shortcut issue. The authors may follow their examination framework to audit the proposed benchmark.

Overall, the target problem and the proposed dataset are important and novel. My rating is weak acceptance at this stage. My final recommendation would depend on the rebuttal.

[r1] Zhong, Jike, et al. "From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning." ICML, 2026.

Questions for authors

See weaknesses.

(Minor) 1. The font sizes in the figures are too small. Furthermore, the language and sentence patterns of this paper are too AI-generated.

(Minor) 2. Missing discussion of several recent related ToM works, such as [r2, r3].

[r2] Zhang, Shunchi, et al. "Mindzero: Learning online mental reasoning with zero annotations." ICML, 2026.

[r3] Zhang, Chunhui, et al. "Overcoming multi-step complexity in multimodal theory-of-mind reasoning: A scalable bayesian planner." ICML, 2025.

5 comments
AuthorsOpenReview ↗

Replies to Reviewer rMk9 ——Weakness 1

Thank you for your time, careful review of our paper. Here are our detailed responses:


W1 Novelty of PEAR.

We thank for this thoughtful comment. We fully understand your concern. We would like to clarify the motivation behind PEAR and contributions by drawing on the core challenges revealed in this study.

As mentioned at the end of the second paragraph and the third paragraph in Chapter 1, Introduction, line 53 to line 91, recent studies have shown that audio LLMs remain strongly biased toward lexical semantics, often relying on verbal content even when vocal expression conveys conflicting information (Chen et al., 2026; Corrêa et al., 2026). At the same time, as shown in Experiment 1&2, our extensive experiments on Overtone involving 16 state-of-the-art speech LLMs and Omni LLMs revealed limitations in their ability to utilize vocal cues, mental-state consistency, and integrate multimodal information.

Inspired by the above, we cannot help but wonder how the human brain integrates and utilizes multimodal information cues to perform ToM reasoning, and whether a mechanism that simulates the human brain’s ToM reasoning process could be used to address these shortcomings. To this end, we propose PEAR.

In the process of ToM reasoning in the human brain, neural information processing is not a passive response to external stimuli, but rather a dynamic predictive coding process. Based on the current context, the brain forms predictions about others’ potential mental states and their likely behaviors; the mismatch between these predictions and observations creates a prediction error, which serves as a signal for adjusting the current mental interpretation, thereby enabling dynamic inference about others’ mental states (Koster-Hale & Saxe, 2013).

While PEAR does not introduce a new model architecture, our goal is to draw inspiration from the way the human brain engages in ToM reasoning, thereby offering a new perspective on how Omni models can integrate information to achieve ToM reasoning. We hope this research will contribute to advancing this field. In future research, we will also explore how to model the human brain and incorporate cognitive mechanisms into model architecture and training objectives.

This response was written word for word with sincerity; it reflects our true motivations and ideas behind proposing the PEAR methodology. This is why our response is somewhat lengthy. Thank you for your patience in reading this; We hope it addresses your concerns. We will provide detailed responses to the remaining questions one by one later. Thank you once again for your valuable feedback.


Reference

[1] Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner. Do audio LLMs really LISTEN, or just transcribe? measuring lexical vs. acoustic emotion cues reliance.

[2] Pedro Corrêa, João Lima, Victor Moreno, Lucas H. Ueda, and Paula Dornhofer Paro Costa. Evaluating emotion recognition in spoken language models on emotionally incongruent speech.

[3] Jorie Koster-Hale and Rebecca Saxe. Theory of mind: A neural prediction problem.

AuthorsOpenReview ↗

Replies to Reviewer rMk9 ——Weakness 2–part (1)

Thank you for your time, careful review of our paper. Here are our detailed responses:


W2 Shortcut audition of Overtone.

Thank you for your expert advice and the reference you provided; These have been crucial in advancing research in ToM field. We have carefully read this paper, learnt from their shortcut audition method, and applied it to Overtone. The detailed methodology and results are as follows:

We use scripts as the basic unit of classification; In each round, we select 6 scripts from each of the 8 ToM task dimensions, perform 12 random restarts per round, with 2,500 steps of local swap steps each time, and remove the scripts selected in each round from the candidate pool for the next round.

Table A: Round-wise Sampling and Ability Distribution of the Discovery Set.

RoundScriptsAudio RecordingsQuestionsQuestions per Each of the 7 ToM AbilitiesScenarios
1485421629–3342
2485120426–3238
3485923630–3738
Total144164656—81 distinct

We first use 164 audio recordings to idnetify rules, then we exclude those scripts and recordings sharing scenes with them. The remaining 1162 recordings are used to evaluate the extracted vocal-cue rules. We use Gemini 3.7 Flash to assessment, we formulate five types of vocal-cue rules based on professional framework for vocal perception( Juslin & Laukka (2003); Anikin & Persson (2018); Bryant (2023); Liebenthal et al. (2016) ), as detailed below:

Table B: Rules for Vocal-Cue Shortcut Auditing.

RulesCues
EmotionWhether positive, negative, neutral, or uncertain vocal labels match the corresponding emotional attitude terms in the answer.
Non-verbal VocalizationsWhether laughter, sighs, and hesitation correspond to options such as joking, disappointment, or uncertainty, respectively.
IronyWhen irony is identified, whether the model directly selects an option corresponding to a discrepancy between literal meaning and intended meaning.
Cue IntegrationSumming the matching of the above emotion, non-verbal vocalization, and irony cues.
Non-literal CommunicationWhen the literal meaning of an utterance is positive but the vocal expression is negative or ironic, prioritizing options indicating surface agreement but underlying dissatisfaction.

Experiment A: LLM-guided Rule Discovery.

Results: Under these rules, no obvious shortcuts for Overtone have been detected.

Table C: Accuracy of Vocal-Cue rules on Overtone.

RulesAccuracyGain
Emotion29.69%+1.68 pp
Non-verbal Vocalizations26.89%−1.12 pp
Irony28.06%+0.04 pp
Cue Integration31.00%+2.99 pp
Non-literal Communication32.64%+4.63 pp

In the 4,648 questions following the isolation scenario, the baseline was 28.01%. Even though these showed an improvement, they fell far short of the 20% threshold specified in the paper. In summary, Overtone passed the shortcut audition.

Experiment B: Lexical Association.

Results: We also calculated the normalized mutual information between surface features and the correct answer. The isolated scenario validation set comprises 4,648 questions and 18,592 candidate options; features include option length, lexical overlap between options and the background, question stem or target sentence, as well as negation, contrast, cognitive and emotional words. The average score across the 11 features was 0.00912, which is below the 0.15 reference threshold proposed in the paper.

Table D: Surface Feature Association Analysis.

Surface FeatureNormalized Mutual Information
Option Word Count0.01070
Option Character Count0.01122
Option–Context Word Overlap0.02319
Option–Question Stem Word Overlap0.00287
Option–Target Utterance Word Overlap0.00190
Contains Negation0.00690
Contains Contrastive Words0.03935
Contains Cognitive Words0.00242
Contains Emotion Words0.00098
Contains Certainty Words0.00077
Mentions Speaker0.0000013
Avg0.00912

Due to space constraints, the remainder part of Weakness2 will be continued in the next comment. Thank you for your patience in reading this.


Reference.

[1] Juslin P N, Laukka P. Communication of emotions in vocal expression and music performance: Different channels, same code?

[2] Anikin A, Bååth R, Persson T. Human non-linguistic vocal repertoire: call types and their meaning.

[3] Bryant G A, Colston H L, Gibbs R W. Vocal strategies in verbal irony.

[4] Liebenthal E, Silbersweig D A, Stern E. The language, tone and prosody of emotions: neural substrates and dynamics of spoken-word emotion perception.

AuthorsOpenReview ↗

Replies to Reviewer rMk9 ——Weakness 2–part (2)

Thank you for your time, careful review of our paper. Here are our detailed responses:


W2 Shortcut audition of Overtone-part (2)

Experiments A and B demonstrate that Overtone passed the shortcut audition proposed in this paper. Furthermore, we also implemented rigorous data design and quality control measures during the construction of Overtone to mitigate potential shortcut risks. Specifically:

  1. Semantically neutral design: Both the background context and target utterances are design in semantically neutral phrasing, avoiding explicit emotional word or direct mental-state cues, thereby preventing the model from directly inferring the answer based solely on emotional keywords in the text. All options are designed to be plausible within the given scenario, such that the correct answer can only be selected based on the voice. See Section 3, lines 154–156 and 208–209.

  2. Controlled voice comparison: We designed shared scripts, strictly maintaining consistency in textual content and social context whilst varying only the voice performances recorded by human. This prevents textual differences from serving as a direct basis for distinguishing mental states, thereby testing whether the model can genuinely infer character’s mental state from vocal information. See Section 3, lines 202–205.

  3. Expert-guided design and review: To avoid templated patterns introduced by LLM-generated content, we had experts in psychology and pragmatics manually construct the scenario families and review the scripts. We also conducted manual listening review of the recordings to ensure rigorous quality control throughout benchmark construction. See Figure 2 (b), Section 3, lines 197–200, and Appendix C.

Furthermore, the results of the Shared Pairs experiment in Table 2 demonstrate that the model is unable to draw correct inferences based on changes in vocal expression; this sharp decline also highlights the diagnostic value of the Overtone, revealing limitations in vocal-cue utilization that may be obscured by overall accuracy. These findings further reveal that the complex patterns underlying human vocal expression are difficult to capture and reproduce through simple, fixed rules.

Reviewer_yySkOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark to examine theory-of-mind (ToM), the ability to perceive belief, desire of others in a human-AI interaction on voice

Towards that end, the authors construct Overtone, the first multimodal ToM benchmark to systematically assess how models use vocal and pragmatic cues to infer mental states. The benchmark has 2,335 social-interaction scenarios with human-recorded speech encompassing diverse ToM tasks. The authors conducted extensive experiments to demonstrate the performance of their benchmark.

Critical strengths

-Proposes a framework for a benchmark to assess vocal and pragmatic cues to infer ToM.

  • The benchmark has human-recorded clips and expert annotated labels for ToM as described in Appendix C.3

-The authors introduce a brain-inspired, training -free framework, PEAR to combine top-down prediction, error feedback, and mental state modeling.

Critical weaknesses
  • The inputs are only text based and evaluations in Section 3.2 compares text and text+audio only. Visual input could strengthen the study.
Questions for authors

Could the authors clarify how many vocal cues did the experts annotate? The authors can find the following framework useful to annotate user perception from voice.

Reference Stack, C.H., Farhana, E., Shen, X., Zhao, S., Maliakal, A., Rashedi, R., Michelson, J. and Kunda, M., 2022. Framework for a multi-dimensional test of theory of mind for humans and ai systems. In The Tenth Annual Conference on Advances in Cognitive Systems.

3 comments
AuthorsOpenReview ↗

Replies to Reviewer yySk ——Weakness 1 & Question 1

Thank you for your time, careful review of our paper. Here are our detailed responses:


W1-1 The inputs are only text based and evaluations in Section 3.2 compares text and text+audio only.

Thank you very much for your feedback. Our benchmark input primarily consists of audio data recorded by real people and text prompts reviewed by experts in psychology and pragmatics.

For the benchmark model evaluation, direct and ToM CoT are evaluated under Text-only, Audio-only, and Audio+Text settings, which differ only in the modality of the target utterance. PC ToM and PEAR use Audio+Text, as both reason over linguistic and vocal information. See Section 5.2, Setting (4).

We will revise the relevant wording in the paper to emphasize the points you raised and avoid any misunderstandings.


W1-2 Visual input could strengthen the study.

Thank you for your suggestion.We agree that visual signals such as facial expressions, gaze, and gestures, provide important information for human social understanding. Overtone is specifically designed to investigate vocal and pragmatic information in human conversational ToM, focusing on how such information shapes the inference of speakers’ mental states. Overtone is not intended to replace broader assessments of theory of mind in audio and video contexts, but rather focuses on a complementary dimension of social understanding that has not yet been fully explored.

We agree that incorporating visual cues into future benchmark expansions to further study cross-modal interaction is a valuable research direction. We will explore this further in subsequent research.


Q1

We thank the reviewer for recommending this relevant work. We have incorporated this work into our discussion of vocal information in ToM assessment. See 2 related work highlighted in blue. Their MD-ToM framework organizes ToM into three domains: perception, knowledge, and reasoning, and they further identify 19 subdomains within these domains. This work provides a relevant theoretical reference for situating vocal and linguistic information within a broader account of ToM.

In Overtone, following (Chevallier et al., 2011), we distinguish vocal expressions by the degree of mental-state inference involved in their interpretation. Basic emotions can often be recognized from vocal patterns with less reasoning about a speaker’s mental states, whereas social emotions and higher-order pragmatic attitudes require perspective-taking, social evaluation, and mental-state attribution. Accordingly, Overtone organizes vocal expressions into four categories: Basic Emotion, Social Emotion, Higher-order Pragmatic Attitudes, and Mixed Social Emotion, covering 19 fine-grained types.

Based on this categorization, after constructing the script, we provide guidance on vocal expressions in human audio recordings across multiple dimensions, including pitch, volume, speaking rate, and so on, as well as guidance on nonverbal vocalizations when appropriate. Finally, experts conduct a review, focusing primarily on the consistency between the predefined affecitve-pragmatic categories and the actual vocal expressions.


References

[1] Stack, C.H., Farhana, E., Shen, X., Zhao, S., Maliakal, A., Rashedi, R., Michelson, J. and Kunda, M., 2022. Framework for a multi-dimensional test of theory of mind for humans and ai systems. In The Tenth Annual Conference on Advances in Cognitive Systems.

[2] Chevallier, C., Noveck, I., Happé, F., & Wilson, D. (2011). What's in a voice? Prosody as a test case for the Theory of Mind account of autism.

Thank you very much to the authors for the revisions and clarifications. This work has the potential to design human-centered AI systems by examining the ToM in voice--- an area still underexplored in the research community and AI system design.

The evaluation could be strengthen by incorporating facial and visual cues. Still, the benchmark is the first step towards prosody research direction and overtone is an important component in prosody research.

Dear Reviewer yySk,

We sincerely thank you for recognizing the contributions and significance of our work. We truly appreciate your positive assessment and thoughtful feedback. Your encouragement means a great deal to us.

Best,
Authors of Submission 31327.

Reviewer_YGRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Overtone, an audio-and-text benchmark for inferring speakers’ beliefs, intentions and emotional attitudes, alongside the training-free PEAR reasoning method. Although the reported results are encouraging, my main concern is the exclusion of facial expressions, gaze and gestures, despite broader claims about human-like social understanding.

Critical strengths

The main strength of the paper is the Overtone dataset, which combines human-recorded speech with textual context and questions about speakers’ attitudes, beliefs and intentions. Its paired examples use the same words spoken differently to convey different meanings, providing a controlled way to evaluate whether models use vocal cues beyond the transcript.

Critical weaknesses
  1. My main concern is that the benchmark considers only audio and text, without facial expressions, gaze or gestures. Facial muscle movements associated with expressive patterns can provide important cues for interpreting affective states. A voice-focused dataset is reasonable, but the paper should better explain what this setting contributes beyond existing visual or audiovisual approaches. What distinct information do vocal cues provide, and in which practical situations is this restricted setting particularly useful?
  2. Figure 3 nevertheless depicts visual information, making the scope of the evaluated framework unclear. The results support reasoning from voice and written context, while the broader claims about human-like social understanding require further evidence. The human baseline also reflects performance under the same restricted conditions. Could the authors clarify which visual components are actually evaluated, distinguish the conceptual framework from its tested implementation, and align the figure and claims with the experimental evidence?
  3. The proposed method is evaluated with audio and text, but its performance with text alone is not reported. Consequently, the gains could partly reflect better general reasoning rather than improved interpretation of vocal cues. The authors should test PEAR with text alone, mismatched audio and neutralized vocal delivery, and report results separately for the controlled vocal contrasts.
  4. A compute-matched baseline is included, but the paper does not report the model-call counts, token budgets or runtime needed to verify that matching. Can PEAR outperform a simpler repeated-reasoning or self-refinement baseline under the same computational budget? This matters when attributing improvements to the proposed mechanism
Questions for authors

Please refer to the weaknesses section

4 comments
AuthorsOpenReview ↗

Replies to Reviewer YGRz ——Weakness 1 & 2

Thank you for your time, careful review of our paper. Here are our detailed responses:


W1 Useful situation and distinct information of vocal cues.

We thank the reviewer for this insightful comment. We fully agree that facial expressions, gaze, and gestures provide important information for understanding others' mental states. However, our intention is not to suggest that vocal cues replace visual signals, but to investigate a complementary and comparatively underexplored source of social information: the vocal and pragmatic information embedded in human conversations. Unlike lexical content, vocal expression conveys rich paralinguistic information through prosody, stress, pauses, non-verbal vocalizations and so on. These clues contain a wealth of information about human social interactions. This setting is particularly relevant to voice-based interactions, including telephone conversations, voice assistants, and audio-only online communication, where visual information is unavailable or inaccessible. We therefore view Overtone as a complementary benchmark targeting a distinct dimension of social understanding. We acknowledge the value of integrating visual signals and will further explore cross-modal social reasoning in future work.


W2 Evaluated visual components and clarify of Figure 3.

We thank the reviewer for pointing out this ambiguity. We would like to clarify the distinction between our framework and its evaluation.

First, Figure 3 illustrates the cognitive motivation of PEAR, which draws on how the human brain integrates multiple sources of social information during ToM inference. Our evaluation of PEAR is not limited to Overtone. We further examine its generalizability across three external ToM benchmarks: MuMA-ToM, MMToM-QA, and Hi-ToM. As the MuMA-ToM and MMToM-QA include visual modalities, so we depict visual information in Figure 3. See Section 5.3: Experiment 3.

For the Overtone model evaluation, direct and ToM CoT are evaluated under Text-only, Audio-only, and Audio+Text settings, which differ only in the modality of the target utterance. PC ToM and PEAR use Audio+Text, as both reason over linguistic and vocal information. See Section 5.2, Setting (4).

Overall, we have proposed the Overtone audio benchmark and the PEAR method, and have validated the effectiveness of PEAR on cross-modal benchmarks that incorporate visual information.

Regarding the human baseline, we agree that it reflects human performance under the same conditions as the Overtone model evaluations. Its purpose is to provide a controlled comparison between human and model performance, rather than to characterize comprehensive human social understanding across all modalities.

We will revise the relevant wording in the paper to emphasize the points you raised and avoid any misunderstandings.


Due to space constraints, we will respond to weaknesses 3 & 4 in the next comment. Thank you for your patience in reading this.

AuthorsOpenReview ↗

Replies to Reviewer YGRz ——Weakness 3

Thank you for your time, careful review of our paper. Here are our detailed responses:


W3 Vocal-Cue Utilization and Performance Attribution of PEAR.

We thank the reviewer for this insightful suggestion. We would like to clarify that, although our benchmark evaluation includes Text-only, Audio-only, and Audio+Text conditions, the PEAR evaluation focuses on Audio+Text, as its prediction-error updating mechanism is designed to operate on vocal observations, This method designed to integrate multimodal information cues, only by observing multimodal information can the prediction error be updated.

To verify the performance improvements in PEAR, we examines from several complementary perspectives.

Experiment A. Has PEAR enhanced the model’s ToM reasoning ability through a more complex reasoning process?

We design a compute matched control, which retains the iterative decision-making process whilst removing the predictive coding mechanism, whilst maintaining an inference budget similar with PEAR, in order to rule out any potential gains arising from the additional computation. PC ToM follows a prediction–comparison–error–update process.

Table A: Performance Comparison of PEAR, PC ToM, and Compute-Matched.

MethodAvg (%) ↑FP (%) ↑
PC ToM76.3347.18
Compute Matched80.5950.14
PEAR87.7167.50

The results shows that Compute Matched remains significantly weaker than PEAR under comparable inference budgets; the results from PC Tom show that simply having the model perform more complex inferences actually introduces more error. These indicating that the performance improvement does not stem solely from additional computation, but primarily from the synergy between the mechanisms.


Experiment B. Has Pear really make use of voice information?

To determine whether PEAR actually makes use of voice information, our evidence for PEAR utilization of voice information extends beyond improvements in overall accuracy. Specifically, we design Shared Pairs evaluation to examines controlled contrasts in which the social context and utterance text remain identical while vocal expression varies.

Table B: Model Performance on Overtone Shared Pairs.

ModelPairAcc (%) ↑Flip (%) ↑
GPT-Audio-1.53.0629.68
Audio-Flamingo-Next8.2422.30
MOSS-Audio-8B5.1227.50
Step-Audio-R1.56.8520.04
Kimi-Audio14.3028.51
MiMo-Audio5.5219.71
Audio-Reasoner1.1312.88
Gemini-3.7-Flash9.8450.00
Gemini-3.1-Pro9.1858.72
Qwen3.5-Omni-Plus3.3245.87
MiMo-V2.57.2530.11
MiniCPM-o-4.51.6029.27
Gemma1.0615.09
Nemotron-3-Nano-Omni5.8523.10
HumanOmni V24.6522.01
OmniSapiens 2.02.8616.93
PEAR23.3467.50

Table C: Comparison of Reasoning Methods on Overtone Shared Pairs.

BackboneMethodPairAcc (%) ↑Flip (%)↑
GPT-Audio-1.5Direct3.0629.68
ToM CoT6.9843.39
PEAR10.3747.27
PC-ToM3.7221.96
Qwen3.5-Omni-PlusDirect3.3245.87
ToM CoT4.9256.06
PEAR17.4262.53
PC-ToM4.4539.41
Gemini-3.1-ProDirect9.1858.72
ToM CoT12.5060.06
PEAR23.6766.67
PC-ToM10.1154.87
Gemini-3.7-FlashDirect9.8450.00
ToM CoT11.1754.02
PEAR23.3467.50
PC-ToM9.4447.18

These results shows that PEAR can use differences in vocal expression to distinguish mental states under identical textual conditions. In particular, PairAcc assesses whether PEAR correctly interprets both vocal realizations, while FP assesses the reliability of its prediction changes. Our evaluation for vocal-cue utilization therefore extends beyond overall accuracy to performance on controlled vocal contrasts.

Furthermore, our ablation studies examine the mechanisms supporting these improvements. Importantly, these ablations retain the audio input while modifying how observations inform mental-state inference. The resulting performance reductions support the contribution of PEAR’s prediction, comparison, updating, and regulation mechanisms to more reliable responses to vocal variation. See Section 5.4: Experiment 4.

Thank you for your patience in reading this; We hope it addresses your concerns. We will provide detailed responses to the remaining question later. Thank you once again for your valuable feedback.

AuthorsOpenReview ↗

Replies to Reviewer YGRz ——Weakness 4

Thank you for your time, careful review of our paper. Here are our detailed responses:


W4-1. Model-call Counts, Token Budgets of PEAR.

We thank the reviewer for emphasizing the importance of a verifiable computational comparison. We agree that attributing performance improvements to the proposed mechanism requires its computational budget.

Table A: Model-call counts and token cost on PEAR and Compute-Matched, “K” denotes thousands of tokens.

MethodBackboneAvg. Model-call per QuestionAvg. Tokens per Question (K)
PEARGPT-Audio-1.54.1790.073
PEARQwen3.5-Omni-Plus4.1470.068
PEARGemini-3.1-Pro4.0792.504
PEARGemini-3.7-Flash4.0400.765
Compute-MatchedGemini-3.7-Flash4.0000.781

As shown in the TableA and the results shown in Section 5.4, Table 6, combined with the performance comparison, PEAR achieves 87.71% Avg and 67.50% FP, compared with 80.59% and 50.14% for the Compute-Matched baseline. PEAR’s advantage persists with similar model-call counts and slightly lower token usage. These results support the contribution of its structured reasoning mechanism beyond simply increasing generation length.


W4-2. Can PEAR outperform other baselines under the same or lower computational budge?

We thank the reviewer for this important question. Our evaluation includes both a compute-matched method on Overtone and comparisons with other ToM reasoning methods on three multimodal cross benchmarks.

 1. The compute-matched baseline in Section 5.4, Table 6 specifically examines whether PEAR’s improvements can be reproduced with a comparable inference budget but without its predictive-coding structure, and the results shows that with comparable model-call counts and lower output-token, PEAR has better performance. These results support the contribution of PEAR’s structured reasoning mechanism beyond simply generating longer responses.

 2. As shown in Section 5.3: Experiment 3: Cross-Benchmark Evaluation and Table 5, we additionally compares PEAR with AutoToM, LIMP, SimToM, BIP-ALM on MuMA-ToM, MMToM-QA, and Hi-ToM. We use GPT-4o as the backbone and matched information access across methods. Since model-calls have not been reported for other methods, we compare PEAR with AutoToM in Table B.

Table B: Comparison of Average Model-call per question between PEAR and AutoToM.

DatasetPEARAutoToM
MMToM-QA28.9717.60
MuMA-ToM10.1135.08
Hi-ToM2.0136.45

These experiments provide additional evidence of PEAR's applicability across different ToM tasks and contexts through comparisons with existing reasoning frameworks. They also provide complementary evidence relevant to the reviewer's concern about whether performance improvements arise merely from repeated reasoning or additional computational expenditure. By comparing PEAR with other ToM reasoning framework, PEAR achieved better results on a lower budget, shows the effectiveness of its reasoning mechanism rather than limiting to a comparison with conventional prompting baselines.

Thank you for your patience in reading this, We hope it addresses your concerns.


Reference.

[1] Zhang Z, Jin C, Jia M Y, et al. Autotom: Scaling model-based mental inference via automated agent modeling.

AuthorsOpenReview ↗

Revised Manuscript and Addressed Reviewer Concerns

Dear Reviewer YGRz,

We have provided a detailed response to all of your questions and suggestions, and have uploaded a revised manuscript accordingly.

If our responses have addressed your concerns, we would kindly appreciate it if you could consider re-evaluating our score.

Thank you very much for your time and consideration.

Best,
Authors of Submission 31327.

Reviewer_dWknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Human study and evaluations of 16 speech and omni-modal models reveal systematic failures in maintaining coherent mental-state representations and integrating vocal cues. The authors therefore propose PEAR (Prediction-Error-Adaptive Reasoning), a training-free predictive-coding-inspired framework that uses mental-state hypotheses to guide multimodal cue interpretation under metacognitive regulation, improving performance across backbones and generalizing across ToM datasets.

Critical strengths
  1. A meaningful research question. The paper examines how vocal delivery, prosody, and pragmatic cues contribute to mental-state inference, extending evaluation beyond text-based Theory of Mind and speech emotion recognition.

  2. A diagnostic shared-script design. Keeping the context and utterance text fixed while varying vocal delivery provides a useful setting for testing whether models use vocal evidence rather than textual shortcuts.

  3. A carefully designed data construction process. Expert guidance, independent review, disagreement adjudication, human recording, and post-recording label verification provide reasonable safeguards for benchmark quality.

  4. Broad evaluation with consistent reported improvements. The experiments cover 16 models, and PEAR improves performance across four backbones. Component ablations and a compute-matched comparison provide additional evidence of its potential.

Critical weaknesses
  1. Insufficient implementation details for reproducibility. PEAR’s complete prompts, execution procedure, and operational criteria for its gating mechanisms are not sufficiently disclosed, making the method difficult to reproduce and independently evaluate.

  2. The sources of the performance gains warrant further analysis. Although improvements in PairAcc and FP support better performance on controlled vocal contrasts, the absence of a text-only PEAR baseline makes it difficult to distinguish general reasoning improvements from gains enabled by additional vocal information.

  3. EM4 does not directly establish mental-state consistency. EM4 assigns credit to a sample only when all four answers are correct. However, a model can give mutually consistent answers based on an incorrect interpretation and receive an EM4 score of zero. A single incorrect answer also produces a zero score, regardless of whether it contradicts the other answers. Thus, EM4 measures joint correctness without distinguishing incorrect but coherent interpretations from internal inconsistencies. Moreover, human answers are aggregated from three participants per question, so human EM4 does not measure an individual’s cross-question consistency.

Questions for authors
  1. Could the authors provide PEAR’s complete prompts, pseudocode, and an execution example, including the criteria used to trigger and accept revisions?

  2. How does PEAR perform with text-only input? Could the authors compare its improvements over the respective baselines under Text-only and Audio+Text conditions to clarify the sources of the performance gains?

  3. Could the authors report an explicit cross-question contradiction rate, assessed separately from answer correctness? This could measure the proportion of scenarios containing at least one pair of mutually incompatible answers, accounting for differences in character, time, and information access. For example, attributing a belief to a character in one answer while explicitly denying that same belief at the same moment in another would constitute a contradiction. This would help distinguish inconsistent mental-state representations from coherent but incorrect interpretations. Could the authors also report individual human accuracy alongside the results obtained by aggregating three participants’ judgments?

8 comments
AuthorsOpenReview ↗

Replies to Reviewer dWkn ——Weakness 3 & Question 3

Thank you for your time, careful review of our paper. Here are our detailed responses:


Q3 Cross-question contradiction rate and Individual human performance.

Q3-1. Cross-question contradiction rate.

We thank the reviewer for highlighting this important distinction. We will assess the semantic compatibility of the four selected answers associated with each audio realization, separately from their correctness. A sample will be counted as contradictory if at least one answer pair expresses mutually incompatible propositions after accounting for the relevant character, time, information access, and perspective. For example, expressing agreement outwardly while experiencing dissatisfaction internally will not be treated as a contradiction. The contradiction rate will be calculated as the proportion of evaluated samples containing at least one such pair. We will report this metrics to distinguish incorrect answers without identified contradictions from internally conflicting interpretations.

We applied this analysis to all 16 models with three input modalities, the specific results are as follows:

Table A: Cross-Question Contradiction Rates across Models and Input Modalities.

ModelText-onlyAudio-onlyAudio+Text
GPT-Audio-1.50.1502%0.1877%0.0751%
Audio-Flamingo-Next0.0000%0.1126%0.0375%
MOSS-Audio-8B0.1126%0.0751%0.0751%
Step-Audio-R1.50.1502%0.1126%0.3003%
Kimi-Audio0.0375%0.1877%0.0751%
MiMo-Audio0.1502%0.0751%0.1502%
Audio-Reasoner0.0375%0.0375%0.0751%
Gemini-3.7-Flash0.0751%0.0751%0.0751%
Gemini-3.1-Pro0.0375%0.0375%0.0375%
Qwen3.5-Omni-Plus0.0000%0.2252%0.0375%
MiMo-V2.50.1126%0.2252%0.1126%
MiniCPM-o-4.50.0751%0.1877%0.1126%
Gemma0.0751%0.0375%0.0751%
Nemotron-3-Nano-Omni0.1126%0.0375%0.0375%
HumanOmni V20.0000%0.0751%0.0000%
OmniSapiens 2.00.1126%0.0751%0.0375%
PEAR----0.0000%

As shown in Table A, contradiction rates range from 0.0000% to 0.3003%. Even the highest rate corresponds to only eight contradictory samples out of 2,664. These indicate that the model rarely produces clear logical conflicts across the four questions corresponding to the same audio.


Q3-2. Individual human performance.

We agree that the aggregated human results should be distinguished from individual performance. We will make this distinction explicit and additionally report accuracy over all unaggregated human responses.

We report the accuracy of all 48 participants individually in Table B, each participant answered 666 questions.

Table B: Individual Accuracy of 48 Human Participants.

Human IDAccuracyHuman IDAccuracy
H00188.59%H02589.64%
H00294.14%H02692.49%
H00385.74%H02789.49%
H00496.10%H02890.84%
H00588.89%H02991.59%
H00690.09%H03094.29%
H00793.24%H03193.54%
H00891.14%H03294.29%
H00995.05%H03392.64%
H01091.29%H03496.70%
H01187.69%H03590.99%
H01285.14%H03692.34%
H01389.94%H03792.04%
H01492.19%H03892.04%
H01587.84%H03988.59%
H01690.09%H04093.24%
H01785.74%H04196.55%
H01888.29%H04290.84%
H01991.44%H04394.59%
H02090.09%H04489.94%
H02189.19%H04588.89%
H02293.69%H04696.10%
H02395.20%H04792.79%
H02494.89%H04889.04%

Thank you for your patience in reading this; We hope it addresses your concerns. We will provide detailed responses to the remaining questions one by one later. Thank you once again for your valuable feedback.

AuthorsOpenReview ↗

Replies to Reviewer dWkn ——Weakness 2 & Question 2

Thank you for your time, careful review of our paper. Here are our detailed responses:


Q2 Vocal-Cue Utilization and Performance Attribution of PEAR.

We thank the reviewer for this insightful suggestion. We would like to clarify that, although our benchmark evaluation includes Text-only, Audio-only, and Audio+Text conditions, the PEAR evaluation focuses on Audio+Text, as its prediction-error updating mechanism is designed to operate on vocal observations, This method designed to integrate multimodal information cues, only by observing multimodal information can the prediction error be updated.

To verify the performance improvements in PEAR, we examines from several complementary perspectives.

Experiment A. Has PEAR enhanced the model’s ToM reasoning ability through a more complex reasoning process?

We design a compute matched control, which retains the iterative decision-making process whilst removing the predictive coding mechanism, whilst maintaining an inference budget similar with PEAR, in order to rule out any potential gains arising from the additional computation. PC ToM follows a prediction–comparison–error–update process.

Table A: Performance Comparison of PEAR, PC ToM, and Compute-Matched.

MethodAvg (%) ↑FP (%) ↑
PC ToM76.3347.18
Compute Matched80.5950.14
PEAR87.7167.50

The results shows that Compute Matched remains significantly weaker than PEAR under comparable inference budgets; the results from PC Tom show that simply having the model perform more complex inferences actually introduces more error. These indicating that the performance improvement does not stem solely from additional computation, but primarily from the synergy between the mechanisms.


Experiment B. Has Pear really make use of voice information?

To determine whether PEAR actually makes use of voice information, our evidence for PEAR utilization of voice information extends beyond improvements in overall accuracy. Specifically, we design Shared Pairs evaluation to examines controlled contrasts in which the social context and utterance text remain identical while vocal expression varies.

Table B: Model Performance on Overtone Shared Pairs.

ModelPairAcc (%) ↑Flip (%) ↑
GPT-Audio-1.53.0629.68
Audio-Flamingo-Next8.2422.30
MOSS-Audio-8B5.1227.50
Step-Audio-R1.56.8520.04
Kimi-Audio14.3028.51
MiMo-Audio5.5219.71
Audio-Reasoner1.1312.88
Gemini-3.7-Flash9.8450.00
Gemini-3.1-Pro9.1858.72
Qwen3.5-Omni-Plus3.3245.87
MiMo-V2.57.2530.11
MiniCPM-o-4.51.6029.27
Gemma1.0615.09
Nemotron-3-Nano-Omni5.8523.10
HumanOmni V24.6522.01
OmniSapiens 2.02.8616.93
PEAR23.3467.50

Table C: Comparison of Reasoning Methods on Overtone Shared Pairs.

BackboneMethodPairAcc (%) ↑Flip (%)↑
GPT-Audio-1.5Direct3.0629.68
ToM CoT6.9843.39
PEAR10.3747.27
PC-ToM3.7221.96
Qwen3.5-Omni-PlusDirect3.3245.87
ToM CoT4.9256.06
PEAR17.4262.53
PC-ToM4.4539.41
Gemini-3.1-ProDirect9.1858.72
ToM CoT12.5060.06
PEAR23.6766.67
PC-ToM10.1154.87
Gemini-3.7-FlashDirect9.8450.00
ToM CoT11.1754.02
PEAR23.3467.50
PC-ToM9.4447.18

These results shows that PEAR can use differences in vocal expression to distinguish mental states under identical textual conditions. In particular, PairAcc assesses whether PEAR correctly interprets both vocal realizations, while FP assesses the reliability of its prediction changes. Our evaluation for vocal-cue utilization therefore extends beyond overall accuracy to performance on controlled vocal contrasts.

Furthermore, our ablation studies examine the mechanisms supporting these improvements. Importantly, these ablations retain the audio input while modifying how observations inform mental-state inference. The resulting performance reductions support the contribution of PEAR’s prediction, comparison, updating, and regulation mechanisms to more reliable responses to vocal variation. See Section 5.4: Experiment 4.


Thank you for your patience in reading this; We hope it addresses your concerns. We will provide detailed responses to the remaining question later. Thank you once again for your valuable feedback.

AuthorsOpenReview ↗

Replies to Reviewer dWkn ——Weakness 1 & Question 1

Thank you for your time, careful review of our paper. Here are our detailed responses:


Q1 PEAR's Reproducibility.

We thank the reviewer for this constructive suggestion. We agree that complete implementation details are necessary for reproducing. We had write the Reproducibility Statement for our paper. We make extensive efforts to ensure the reproducibility of this work. We will release the Overtone benchmark resources, PEAR's code, and evaluation scripts alongside the publication of this work to facilitate independent verification and future research.

Trigger Revision: PEAR initiates further reasoning when unresolved mental-state competitors remain; otherwise the initial interpretation H0H_0 is retained. See Section 4.2, lines 273-275 and Appendix H.

Accept Revision: A candidate revision H1H_1 is accepted only if it passes the metacognitive stability check; otherwise H0H_0 is retained. See Section 4.3 and Appendix H.


Thank you for your patience in reading this, We hope it addresses your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.