acceptodds
Under review as a conference paper at ICLR 2027

CONTACT: A Human-Grounded Benchmark and Surprisal-Based Predictive Scorer for Conversational Naturalness

11 views

Abstract

Spoken dialogue evaluation has advanced from response correctness to turn- taking, emotional responsiveness, and conversational quality. Yet a central chal- lenge remains: determining whether automatic scores reflect the naturalness peo- ple experience across both targeted interaction failures and spontaneous conver- sation. We introduce CONTACT, a human-grounded benchmark and a predictive scorer to address this challenge. The benchmark combines live human–human conversations with elicited turn-taking and affective disruptions, and free-form human–AI dialogues. Human ratings anchor the assessment of turn-taking, af- fective, and overall naturalness across the benchmark’s subsets, connecting diag- nostic control with naturalistic evaluation. Alongside this resource, we propose a surprisal-based scorer trained on approximately 1,600 hours of natural dyadic speech to anticipate future turn-taking and affective states. Surprisal of the ob- served speech given a previous context yields dimension-specific scores and an overall composite without requiring a matched reference conversation. Results suggest that the proposed scorer provides a higher correlation with the perceived naturalness provided by the benchmark than current full-duplex evaluation sys- tems. Together, the benchmark and scorer connect targeted diagnosis, human- perceived interaction quality, and natural-dialogue predictive learning, providing a reusable basis for evaluating conversational naturalness.

Primary area:
datasets and benchmarks
Keywords:
conversational naturalness, spoken dialogue evaluation, human-AI interaction, turn-taking, affective response, human-grounded benchmarks, predictive learning, surprisal

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PkZtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies conversational naturalness in multi-turn spoken dialogue, focusing on turn-taking and affective appropriateness. It introduces CONTACT, a benchmark containing 493 conversations across three subsets: human–human conversations with elicited turn-taking disruptions, human–human conversations with elicited affective mismatches, and open-ended human–AI conversations. The first two include natural baseline conversations alongside the manipulated conditions, while the third examines spontaneous interactions with spoken dialogue systems. Participants and recording supervisors provide the main human ratings, assessing the corresponding interaction dimension in each human–human subset and turn-taking, affective appropriateness, and overall naturalness in human–AI conversations.

The authors also propose a surprisal-based scorer that predicts future joint voice activity, joint silence, and arousal–valence states from preceding dialogue context. The predictor is trained on approximately 1,636 hours of natural human conversation, without human naturalness-label supervision. At evaluation time, activity and silence likelihoods, together with an affective component based on between-speaker differences in predictive gain, are combined into turn-taking and affective scores. Their mean serves as the overall naturalness score. The method requires no matched reference conversation, while score normalization and fusion weights are determined on development data.

Validation examines both whether scores distinguish natural from manipulated conversations and whether they correlate with graded human ratings, using development and test splits reported to be speaker-disjoint. Comparisons include turn-taking predictors, speech-quality and speech-reward models, an affective-entrainment model, and a timing-statistics control. On CONTACT test sets, the proposed scorer obtains higher correlations with combined participant–supervisor ratings than the evaluated baselines in both human–human subsets and across all three human–AI dimensions. Score-component ablations further show contributions from joint-silence scoring to human–human turn-taking-rating alignment and from arousal–valence scoring to human–AI affective-rating alignment.

Critical strengths
  1. This paper provides a distinct perspective on evaluating conversational naturalness by examining both disruptions and alignment with human judgements. CONTACT combines controlled human-human conversations with open-ended human-AI dialogue to assess temporal and affective coordination. Its live recordings preserve the partner's responses and subsequent exchange, which are crucial to evaluating a disruption's effect. This makes CONTACT a substantive evaluation resource for distinguishing sensitivity to disruptions from alignment with perceived conversational quality.
  2. The paper also provides a learning method that explicitly incorporates temporal coordination and affective alignment. Predicting future activity, silence, and arousal-valence states from preceding dialogue addresses the context dependence of conversational appropriateness. Automatically extracted targets allow training on approximately 1,636 hours of natural dialogue, while the smaller CONTACT benchmark supports calibration and human evaluation. Higher correlations with combined human ratings than the evaluated baselines provide empirical support for this approach. The method thus offers a way to learn contextual patterns of interaction at scale and assess their relationship with perceived naturalness, without requiring naturalness ratings throughout the training corpus.
Critical weaknesses
  1. The benchmark's diagnostic conclusions depend on human ratings and elicited behaviours whose reliability is only partly established. The protocol describes instruction and practice, but does not report how consistently the intended behaviours were realized or their frequency and intensity. Differences attributed to disruption type in Section 3.5 may therefore also reflect variation in enactment. The independent audit covers 72 conversations rated by one additional listener (Appendix D.1), providing limited evidence of reproducibility across independent listeners. These gaps limit confidence in CONTACT as a human-grounded reference for comparing naturalness metrics, which is a central contribution of the paper.
  2. The scorer's validation leaves uncertainty about the source of its gains and their practical significance for assessing conversational naturalness. Although Table 4 reports significant improvements, statistically significant gains on some comparisons, the human-AI overall correlation with combined ratings is modest at 0.36. The score-removal ablations assess readout components rather than the contributions of training objectives (Table 4; Appendix E.6), while the criterion for selecting CONTACT dev fusion weights is unspecified (Appendix E.3). This makes it difficult to distinguish the contributions of predictive learning and benchmark-specific calibration. The evidence therefore supports comparative improvement more clearly than it establishes the reliability of the proposed design as a measure of conversational naturalness.
Questions for authors
  1. What perceptual checks confirm that CONTACT recordings convey the intended behaviours or affective expressions? Please report the observed frequency and intensity of the disruptions and affective behaviours, and explain how these support the disruption comparisons.
  2. What scoring criteria distinguish the intermediate ratings of 2–4? How many of the 493 conversations were independently rated by multiple evaluators, and what evidence supports the reliability and representativeness of the evaluated subset?
  3. How are the training objectives implemented across stages? Please provide a table of trainable modules and active losses, and clarify how the fusion weights were selected.
  4. What evidence isolates the contributions of the individual training objectives and CONTACT-dev calibration to test performance? In particular, were retraining and calibration ablations conducted, and how should the reported score-component removal results be interpreted as evidence of component-level contributions?
  5. To what extent do recording-path delay and cross-device alignment affect perceived naturalness and scorer comparisons? Can these effects be distinguished from the intended interaction effects, beyond the separate latency probe?
7 comments
AuthorsOpenReview ↗

Response to Reviewer PkZt (1/4)

We thank the reviewer for the detailed review and the many concrete questions.

W1 / Q1: Were the intended behaviors realized?

The protocol describes instruction and practice, but does not report how consistently the intended behaviours were realized or their frequency and intensity. Differences attributed to disruption type in Section 3.5 may therefore also reflect variation in enactment.

What perceptual checks confirm that CONTACT recordings convey the intended behaviours or affective expressions? Please report the observed frequency and intensity of the disruptions and affective behaviours, and explain how these support the disruption comparisons.

Every retained recording passed recording-time checks, and acoustic measures and blind evaluators' notes confirm that the instructed behaviors occurred.

Recording-time checks. For every manipulated conversation, the research supervisor confirmed that the instructed behavior occurred at least four to five times and was clearly noticeable, and the uninstructed partner, asked afterwards without any hint, noticed a difference from the natural conversation. Conversations that failed either check were recorded again.

Blind evaluators' notes. Because recordings were retained only when the partner noticed a difference, we also checked with listeners outside the recording. The new independent evaluators (see General Response), blind to condition, wrote a reason for every rating. These notes mention the targeted disruption, such as an interruption, a long overlap, an unfinished turn, or flat affect, in 51% of the notes on manipulated H-H conversations, whereas notes on natural conversations mention the same disruptions in only 9% of cases.

Per-condition checks. Table R.7 compares each manipulated conversation with the natural conversation of the same session, so the speakers and recording setup are held fixed.

SubsetConditionAcoustic measureChangeBlind notesPartner MOS change
Awkward Turn-TakingPremature EntryOverlap ratio+.036*12/16 vs. 3/11-1.29
Blocked Invited ResponseInstructed speaker's retakes (per recording)+1.9*4/12 vs. 0/11-1.43
Competitive Floor ConflictOverlap ratio+.124*8/12 vs. 0/11-
Fragmented Turn HoldingInstructed speaker's within-turn pause (s per recording)+25.5*7/12 vs. 2/11-1.86
Unmarked Turn AbandonmentSilences over 2 s (per recording)+8.1*7/11 vs. 0/11-1.96
Affective MismatchOverreactionInstructed speaker's arousal+.115*3/14 vs. 0/13-1.00
Under-ResponseInstructed speaker's arousal-.181*8/13 vs. 3/13-1.92
Valence Mismatch--4/13 vs. 0/13-2.00

Table R.7: Manipulated minus natural conversation of the same session. Blind notes: notes mentioning the targeted disruption, manipulated vs. natural (coded blind to condition by an LLM with a fixed codebook). Partner: the speaker who received no instruction. *: 95% CI excludes zero; all MOS changes are significant (Wilcoxon p<.001p<.001). A retake is the instructed speaker resuming within 1 s after the partner starts. Competitive Floor Conflict instructs both speakers. Definitions, sample sizes, and CIs are in Appendix F.2.

All five turn-taking measures change in the instructed direction, and the instructed speaker's arousal rises in Overreaction and falls in Under-Response. Valence Mismatch has no acoustic entry because it is defined relative to each topic's intended valence, which a recording-level acoustic measure does not capture. It is supported by the recording-time checks, the blind notes, and the largest partner rating drop of any condition.

Revisions. Section 3.2 now describes the recording-time checks. New Appendix F.2 reports Table R.7 with all measures, definitions, and CIs.

AuthorsOpenReview ↗

Response to Reviewer PkZt (2/4)

W1 / Q2: Rating scale, coverage, and the independent audit

The independent audit covers 72 conversations rated by one additional listener (Appendix D.1), providing limited evidence of reproducibility across independent listeners.

What scoring criteria distinguish the intermediate ratings of 2-4? How many of the 493 conversations were independently rated by multiple evaluators, and what evidence supports the reliability and representativeness of the evaluated subset?

Every conversation is rated by both the participant and the supervisor, and 212 conversations are now also rated by new independent evaluators blind to condition.

Scale. Raters used a 5-point scale with labeled endpoints (1 = Very Unnatural, 5 = Very Natural). As in standard MOS protocols [1], the rating is a holistic judgment and no criteria are defined for the intermediate points, which avoids steering raters toward particular behaviors. The new evaluators' written reasons show how the middle of the scale is used in practice. For example, a 3 is often given when one aspect is natural and another is not, such as smooth turn-taking with mismatched emotions.

Coverage. All 493 conversations are rated by the participant and the supervisor. In addition, independent evaluators blind to condition and outside data collection rated 212 conversations (127 H-H and 85 H-AI; I2 in Table R.5), including the 72 rated by the original independent evaluator (I1; Appendix D.1). All independent ratings will be released with the corpus.

Reliability. On the same 72 conversations, using the new independent ratings in place of the original ones raises three-rater agreement on every dimension, and the original and new independent ratings agree with each other at ICC .54 to .76 (Table R.5). Across all 212 conversations, three-rater agreement is moderate on every dimension (ICC .51 to .60), in the same range as the agreement between participant and supervisor on the full corpus (ICC .39 to .70; Table 3 in the manuscript).

RatersConversationsH-H TTH-H Aff.H-AI TTH-AI Aff.H-AI Overall
P-S-I1 (App. D.1)72.70 (79%).52 (50%).61 (58%).42 (54%).57 (58%)
P-S-I2 (new)72.78 (83%).67 (54%).63 (58%).50 (62%).59 (54%)
P-S-I2 (new)212.60 (66%).51 (47%).54 (56%).52 (58%).54 (60%)
I1-I2 (new)72.54 (75%).57 (71%).67 (83%).58 (83%).76 (92%)

Table R.5: Rating agreement, ICC(A,1) with Within-1 agreement in parentheses. P: participant; S: supervisor; I1: original independent evaluation; I2: new independent evaluation. The 72 audited conversations include 24 per subset.

Representativeness. The audited subset covers all conditions and all six H-AI systems (four conversations each), and its Combined-MOS distribution is close to that of the full set, for example 3.04±1.193.04 \pm 1.19 vs. 3.14±1.063.14 \pm 1.06 for H-H turn-taking and 2.71±1.112.71 \pm 1.11 vs. 2.85±1.152.85 \pm 1.15 for H-AI overall.

Revisions. Section 3.4: "Then, an independent evaluator blind to condition re-rated the conversations (Appendix D.1)." -> "Then, independent evaluators blind to condition re-rated a subset of 212 conversations (Appendix D.1)." Appendix D.1 adds Table R.5 and the subset-versus-full comparison.

[1] ITU-T. Recommendation P.800: Methods for subjective determination of transmission quality. 1996.

AuthorsOpenReview ↗

Response to Reviewer PkZt (3/4)

W2 / Q3 / Q4: Training objectives, fusion weights, and calibration

The score-removal ablations assess readout components rather than the contributions of training objectives (Table 4; Appendix E.6), while the criterion for selecting CONTACT dev fusion weights is unspecified (Appendix E.3).

How are the training objectives implemented across stages? Please provide a table of trainable modules and active losses, and clarify how the fusion weights were selected.

What evidence isolates the contributions of the individual training objectives and CONTACT-dev calibration to test performance? In particular, were retraining and calibration ablations conducted, and how should the reported score-component removal results be interpreted as evidence of component-level contributions?

No training stage uses CONTACT, ablations show that both refinement objectives contribute, and the gains hold without any CONTACT calibration.

Training stages (Q3). Table R.8 lists what is trained in each stage. No stage uses CONTACT data or labels.

StageTrainableFrozenObjectivesSteps
1. Warm-start (from public DualTurn)Backbone, input projections, all branches and headsMimi encoderFuture joint voice activity50k
2. Affect refinementAffect pathway and A/V headsBackbone, timing pathwayFuture affective state10k
3. Timing + silence refinementTiming pathway and heads, silence headBackbone, affect pathwayFuture joint voice activity and future joint silence4k

Table R.8: Trainable modules, frozen modules, and active objectives in each training stage.

Fusion weights (Q3). The timing and affective scores each combine two components with one weight. During model development, each weight was selected on development data to maximize the H-AI correlation while keeping H-H discrimination within a tolerance of equal weights, and was then kept fixed. Appendix E.3 now states this, and the calibration ablation below tests equal weights directly.

Training-objective ablations (Q4, new). Stages 2 and 3 each add one objective (Table R.8): affect refinement and the joint-silence loss. Table R.9 removes each in turn.

ModelH-H TT Acc.H-H Aff. Acc.H-AI TT ρ\rhoH-AI Aff. ρ\rho
Full model86.082.4.40.37
w/o joint-silence loss84.060.8.33.33
w/o affect refinement (stage 2)86.078.4.40.33

Table R.9: Training-objective ablations on the test set (Combined MOS for ρ\rho). Rows 1-2: stage 3 retrained for 4k steps with and without the joint-silence loss; without it, the model has no silence score. Row 3: the model trained without stage 2. Because stage 2 updates only the affect pathway and stage 3 neither updates nor uses it, this model combines the affect pathway saved before stage 2 with the full model's activity and silence components.

Removing the joint-silence loss lowers all four measures, most strongly H-H affective accuracy (82.4% to 60.8%). The loss does not change the voice-activity component (84.0% in both runs), so its benefit comes entirely from the silence score it trains, which is why row 2 matches the silence-removal row of Table 4 in the manuscript. Removing affect refinement leaves turn-taking unchanged and lowers the H-AI affective correlation from .37 to .33 (paired 95% CI of the difference [.011,.085][.011, .085]); its H-H change is not significant. For the remaining components, score removal does not isolate a training objective, so we now call these score-component ablations in Section 5.2.

Reviewer_pH2SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors present CONTACT, a benchmark for naturalness in vocal dialogue between two participants. CONTACT includes human-human dialogues as well as human-AI dialogues, with various manipulations included within the dataset (like interruptions, etc.) as well as grades from human labelers. In parallel, authors constructed a naturalness scorer based on loss functions corresponding to future silence, activity, and affective state predictions, and evaluated it against CONTACT. They showed that the scorer performed well in this setting.

Critical strengths

(1) Human naturalness is a difficult quantity to measure and is also a fairly prominent gap in benchmarking in most aspects of GenAI output. Having a vocal benchmark testing this type of metric is useful and fills a critical gap.

(2) Analysis in the paper is fairly thorough and includes ablation studies as well as very detailed descriptions of how all data was collected. Multiple reviewers were used for naturalness labels, with a mixture of the original labelers and new labelers who go in blind. There is not much room for doubt on what the practitioner is getting with this benchmark.

Critical weaknesses

(1) I generally liked this paper. However one of my big concerns is whether the proposed scientific proof of CONTACT's effectiveness is a bit circular. CONTACT is not a huge dataset so I do not expect the authors to train new voice models using CONTACT as a training set, but producing a scoring model in parallel and then testing it on CONTACT seems a bit strange since it is unclear if the scoring model is overfit to CONTACT just based on its loss function design of other aspects.

(2) The manipulations the authors used to augment CONTACT such as premature entry seem very sensible. However, it is unclear why these particular augmentations were selected and whether they are particularly important to the community as a whole. It would be nice if the authors could explain why these set of augmentations are complete.

(3) Annotator agreement seems to only be partially done at time of submission, and so far even on the limited set some agreement scores seem a bit weak like for affective mismatch. Given the subjectivity inherent in benchmarks like this, agreement scores are critical for assessment of the usefulness of the benchmark, and we need more clarity here.

Questions for authors

(1) Can the authors benchmark the scoring model against any additional public benchmarks of audio interaction quality? This way we have a more objective measure of whether the scoring model quality is high.

(2) Can authors provide any samples of the data in a way that does not break double-blindness? It is a bit strange to evaluate a benchmark paper without actually having any access to samples.

(3) Can authors provide complete agreement analysis to quantify intra-rater variance?

(4) Can authors contextualize why the specific dataset augmentations were selected as they were and why they are convinced this is a complete set?

(5) What precisely is the use case for the dataset, if it is not big enough to be reasonably used as a training dataset? Can practitioners use it to build any models or is it primarily an evaluation tool?

I generally liked the paper and am willing to move my score up if adequate answers and additional evidence are provided.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pH2S (1/4)

We thank the reviewer for the encouraging review and the concrete questions.

W1: Circularity and possible overfitting to CONTACT

one of my big concerns is whether the proposed scientific proof of CONTACT's effectiveness is a bit circular. CONTACT is not a huge dataset ... producing a scoring model in parallel and then testing it on CONTACT seems a bit strange since it is unclear if the scoring model is overfit to CONTACT just based on its loss function design of other aspects.

CONTACT is validated by human ratings, not by our scorer. The scorer never sees CONTACT in training, and its gains hold without any CONTACT calibration.

We see why the setup can look circular. Whether CONTACT works as a benchmark is decided by human listeners. On average, every elicited condition is rated less natural than its natural baseline (Table 3 in the manuscript), and in the conditions where only one speaker was instructed, even the uninstructed partner rated the conversation 1.0 to 2.0 points lower than the natural one from the same session (all p<.001p<.001). The scorer is our second contribution, tested on CONTACT in the same way as the baselines, so it plays no part in validating the benchmark and the argument is not circular.

On overfitting, the predictor is trained only on 1,636 h of natural conversations from the Seamless Interaction dataset to predict who will speak when, how long both speakers will stay silent, and the arousal and valence of the responder's voice. The loss function design simply compares these predictions with what actually happens next in Seamless Interaction, without naturalness labels or any CONTACT data. Two further checks support this. First, nothing was manipulated in the H-AI conversations, yet removing the joint-silence or A-V score significantly lowers the H-AI turn-taking (.40 to .33) and affective (.37 to .32) correlations (Appendix E.6 in the manuscript). Second, since CONTACT is not a huge dataset, we took it out of the calibration altogether, using Seamless-only normalization and equal fusion weights (setting (c) in Table R.3).

SettingNormalizationFusion weightsH-H TTH-H Aff.H-AI TTH-AI Aff.H-AI Overall
(a) PaperCONTACT-devDev-selected.36.28.40.37.36
(b) Equal weightsCONTACT-devEqual.35.22.40.42.37
(c) No CONTACT calibrationSeamless onlyEqual.39.22.40.44.39
Strongest baseline--.19.17.35.26.26

Table R.3: Test Combined-MOS Spearman ρ\rho of the same checkpoint under different calibration settings.

Without any CONTACT calibration, results stay the same or improve on four of the five dimensions, and the gain over the strongest baseline remains significant for H-H turn-taking (Δρ=+.196\Delta\rho=+.196, 95% CI [.020,.385][.020, .385]), H-AI affective (+.179+.179, [.023,.341][.023, .341]), and H-AI overall (+.135+.135, [.040,.237][.040, .237]). H-H affective is lower only because the dev-selected weights favor it (.28 vs. .22). The gains therefore come from the predictor rather than from calibration, so the scorer is not overfit to CONTACT.

Revisions. Section 1 now states that the scorer is evaluated on CONTACT in the same way as existing metrics and released as a baseline for future work, and Section 5.2 with new Appendix F.1 reports the calibration-free results (Table R.3).

AuthorsOpenReview ↗

Response to Reviewer pH2S (2/4)

W2 / Q4: Why these conditions, and is the set complete?

it is unclear why these particular augmentations were selected and whether they are particularly important to the community as a whole. It would be nice if the authors could explain why these set of augmentations are complete.

Each condition reproduces a failure that recent evaluations report in current voice agents, and together the conditions cover the main steps of a turn exchange and both dimensions of affect. The set is not exhaustive.

We started from failures that recent evaluations report explicitly. Current voice agents interrupt at unlikely moments, take over during a user's mid-turn pause, keep talking when the user interrupts, and give few cues when they want to keep the floor [1, 2]. Premature Entry, Fragmented Turn Holding, and Competitive Floor Conflict reproduce these failures. For affect, the emotional expression of AI voice models is rated far below that of humans for naturalness and appropriateness [3], their speech is often flat [4], and some systems in our own H-AI sessions sounded overly enthusiastic, which Under-Response and Overreaction reproduce.

The five turn-taking conditions also follow the main steps of a turn exchange in the turn-taking model of Sacks et al. [5]: starting where the turn can change (Premature Entry), handing over a turn once offered (Blocked Invited Response), resolving overlap (Competitive Floor Conflict), holding the turn across pauses (Fragmented Turn Holding), and ending a turn with cues the partner can respond to (Unmarked Turn Abandonment). The three affective conditions cover errors along both dimensions of the arousal-valence model [6]: arousal too high (Overreaction) or too low (Under-Response), and the wrong valence (Valence Mismatch). Table R.4 (new) maps each condition to the failure it reproduces.

SubsetConditionWhat it reproduces in human-AI conversation
Awkward Turn-TakingPremature EntryStarting before the partner has finished, as when an agent cuts in at an unlikely moment [1]
Blocked Invited ResponseOffering the turn with yielding cues [7] and taking it back once the partner starts, as when an agent pauses and then talks over the user
Competitive Floor ConflictLong overlap in which neither side yields, as when an agent keeps talking after the user interrupts [1, 2]
Fragmented Turn HoldingRepeated pauses within a turn that make holding hard to tell from yielding, which current agents misread [1, 2]
Unmarked Turn AbandonmentStopping without finishing or yielding, leaving the partner without cues to respond [7], as when an agent's reply cuts off mid-sentence
Affective MismatchOverreactionAffect stronger than the context calls for, such as excessive enthusiasm
Under-ResponseAffect weaker than the context calls for, such as a flat reply [4]
Valence MismatchAffect of the wrong polarity, as when an agent misreads the user's emotion [8]

Table R.4: The failure each condition reproduces, in the order of the manuscript.

These failures are already central to how the community evaluates voice agents, as full-duplex benchmarks test pause handling, turn transitions, and interruptions [2], which speaks to their importance to the community.

We agree that the set is not complete. We focused on failures that a person can enact reliably within a natural conversation and that listeners clearly notice (Table 3 in the manuscript). Backchannel timing, which current systems also handle poorly [1], and prosodic entrainment are natural next additions.

Revisions. Section 3 now states this selection rationale with the supporting references, new Appendix A adds Table R.4, and Section 6 states that the conditions are not exhaustive.

[1] S. Arora et al. Talking Turns: Benchmarking audio foundation models on turn-taking dynamics. In ICLR, 2025.

[2] G.-T. Lin et al. Full-Duplex-Bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. In IEEE ASRU, 2025.

[3] J. Liu et al. EMO-Reasoning: Benchmarking emotional reasoning capabilities in spoken dialogue systems. In IEEE ASRU, 2025.

[4] X. Luo et al. AEQ-Bench: Measuring empathy of omni-modal large models. In Findings of ACL, 2026.

[5] H. Sacks, E. A. Schegloff, and G. Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696-735, 1974.

[6] J. A. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161-1178, 1980.

[7] A. Gravano and J. Hirschberg. Turn-taking cues in task-oriented dialogue. Computer Speech & Language, 25(3):601-634, 2011.

[8] J. Wagner et al. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 2023.

AuthorsOpenReview ↗

Response to Reviewer pH2S (3/4)

W3 / Q3: Agreement and intra-rater variance

Annotator agreement seems to only be partially done at time of submission, and so far even on the limited set some agreement scores seem a bit weak like for affective mismatch.

Can authors provide complete agreement analysis to quantify intra-rater variance?

Every H-H conversation is rated by both speakers and by the research supervisor, and new blind evaluators show that the ratings largely reproduce across listeners.

All 493 conversations are rated by the participant and the supervisor, whose ratings are within one point of each other for 89% (H-H turn-taking), 80% (H-H affective), and 68-73% (H-AI) of conversations (Table 3 in the manuscript). In addition, independent evaluators blind to condition and outside data collection rated 212 conversations (127 H-H and 85 H-AI), including the 72 rated by the original independent evaluator (Table R.5). Each rating comes with a written justification. All independent ratings will be released with the corpus.

RatersConversationsH-H TTH-H Aff.H-AI TTH-AI Aff.H-AI Overall
P-S-I1 (App. D.1)72.70 (79%).52 (50%).61 (58%).42 (54%).57 (58%)
P-S-I2 (new)72.78 (83%).67 (54%).63 (58%).50 (62%).59 (54%)
P-S-I2 (new)212.60 (66%).51 (47%).54 (56%).52 (58%).54 (60%)
I1-I2 (new)72.54 (75%).57 (71%).67 (83%).58 (83%).76 (92%)

Table R.5: Rating agreement, ICC(A,1) with Within-1 agreement in parentheses. P: participant; S: supervisor; I1: original independent evaluation; I2: new independent evaluation. The 72 audited conversations include 24 per subset.

On the same 72 conversations, with the new independent ratings in place of the original ones, three-rater agreement is higher on every dimension, including H-H affective (.67 vs. .52). The original and new independent ratings also agree with each other, with ICC from .54 to .76 and ratings within one point for 71-92% of conversations. Across all 212 conversations, three-rater agreement is moderate on every dimension (ICC .51 to .60), in the same range as the agreement between participant and supervisor on the full corpus (ICC .39 to .70; Table 3 in the manuscript).

We did not collect repeated ratings from the same rater, so intra-rater reliability cannot be estimated directly, and we list this as a limitation. Our inter-rater analysis instead evaluates how well the ratings reproduce across independent listeners. Affective agreement remains somewhat lower, as judging whether an emotional response fits its context is more subjective than judging timing, which is why we report results against participant, supervisor, and combined ratings separately.

Revisions. Section 3.4 and Appendix D.1 report the new independent evaluation and Table R.5, and Appendix F.5 adds the intra-rater limitation.

Reviewer_21UNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper introduces CONTACT, a benchmark andand automatic scoring method for evaluating naturalness in spoken dialogue. The benchmark consists of human-human (H-H) conversations as well as open-ended human-AI (H-AI) conversations, with human ratings of perceived naturalness. The proposed scorer is trained on natural human conversations to predict future joint voice activity, joint silence, and affective states. It evaluates conversational naturalness based on how well these predictions match observed interactions. Experiments demonstrate improvements over existing baselines in both discriminating unnatural interactions and correlating with human judgements.

Critical strengths
  • The collection of live human-human converstaions with controlled interaction disruptions, alongside naturalistic human-AI conversations, provides a potentially useful resource for evaluating conversational naturalness.
Critical weaknesses
  1. Insufficient justification for the proposed naturalness scoring formulation

    • The main assumption that conversational naturalness corresponds to predictability is not sufficiently justified. Unexpected responses can be natural, while predictable responses can be inappropriate.
    • The authors define affective naturalness as the negative absolute difference between speakers' predictive gains. I think this rewards similarity rather than appropriateness. Two speakers with equally poor predictive gains could receive a higher score than two speakers with strong but unequal gains.
  2. Unclear whether the scorer actually measures affective appropriateness.

    • The proposed affective score assigns 75% of its weight to join silence and only 25% to arousal-valence. Removing arousal-valence barely changes the H-H affective MOS correlation (0.28 to 0.27) whereas removing joint silence substantially reduces affective discrimination accuracy (82.4 to 60.8). These results raise concerns that the scoreer primarily detects temporal disruptions rather than the contextual appropriateness of emotional responses.
Questions for authors

Please see weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 21UN (1/3)

We thank the reviewer for the specific and thoughtful comments. We address each point below.

W1.1: Predictability vs. naturalness

The main assumption that conversational naturalness corresponds to predictability is not sufficiently justified. Unexpected responses can be natural, while predictable responses can be inappropriate.

Our surprisal measures how unlikely a continuation is in natural conversational data, not how unexpected it seems to a listener.

Thank you for this point. We believe the key is to distinguish being unexpected to a listener from being unlikely in natural conversation. Our scorer measures only the latter. The predictor is trained on 1,636 h of natural conversation and, given the dialogue so far, predicts who will speak when, how long both speakers will stay silent, and the arousal and valence of the responder's voice. Surprisal is how unlikely the actual continuation is under these learned predictions, so what counts as expected is determined by unlabeled natural conversations, not by human intuition or rules we set.

This distinction addresses both cases in the comment. A response unexpected to a listener is not penalized as long as the model has learned such conversational behavior from the training data. A response that is common but inappropriate in its context is penalized, because the model predicts from the context, not from overall frequency. A similar idea has been used for sentences, where probabilities from language models trained on natural text predict human acceptability ratings [1].

That said, some natural behaviors are simply rare in the training data and may still receive high surprisal, as the training data does not contain all possible cases. More varied training data should reduce this, and we now list it as a limitation. For the same reason, we test the proxy against human ratings, which is what CONTACT is designed for. Our scorer aligns with human ratings better than all baselines, both on the controlled H-H conversations and on the spontaneous H-AI conversations, where no behavior was manipulated (Table 4 in the manuscript). We have revised Section 4 to state this as a hypothesis.

Revision. Section 4: "We therefore treat naturalness as conformity with context-conditional expectations" -> "We therefore hypothesize that perceived naturalness can be approximated by conformity with context-conditional expectations, and we test this hypothesis against human ratings on CONTACT."

Appendix F.5 (Limitations), added: "The predictor learns what to expect from its training data, so natural but rare behaviors may still receive high surprisal. Training on larger and more varied conversational data should reduce this."

[1] J. H. Lau, A. Clark, and S. Lappin. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5), 2017.

AuthorsOpenReview ↗

Response to Reviewer 21UN (2/3)

W1.2: The affective gain gap

The authors define affective naturalness as the negative absolute difference between speakers' predictive gains. I think this rewards similarity rather than appropriateness. Two speakers with equally poor predictive gains could receive a higher score than two speakers with strong but unequal gains.

Affective mismatch is typically one-sided, with one speaker responding inappropriately to a partner who behaves naturally, and unequal gains capture exactly this.

Thank you for this counterexample. Our design assumes that an affective mismatch is one-sided, with one speaker behaving naturally and the other departing from what the context calls for. This matches how the score is meant to be used. When a spoken dialogue system is evaluated in conversation with a human user, any mismatch is expected to come from the system, while the human user behaves naturally. Both CONTACT subsets follow this setting. In the H-H affective subset, one speaker receives an instruction on how to express their response, while the partner is "asked to respond naturally" (Appendix B.2.2 in the manuscript). In the H-AI conversations, participants were instructed to "chat as you normally would" and to treat the agent as a human partner (Appendix B.2.3 in the manuscript). In these settings, a mismatch shows up as a difference between the two speakers' gains, so a lower score for strong but unequal gains is the intended behavior.

We agree that the score could reward similarity in some cases, as the reviewer describes, where both speakers depart from natural behavior and are poorly predicted. However, this case should be rare when one side behaves naturally. To check how often this happens, we counted the recordings in which both speakers are predicted worse than by the affect prior, that is, both gains are negative (gˉ0<0\bar{g}_0<0 and gˉ1<0\bar{g}_1<0). This count includes the case of equally poor gains, since it does not require the two gains to be equal. Such recordings make up 0 of 100 H-H affective recordings and 4 of 213 H-AI recordings. A correction that targets only this case, for example by lowering the score when both gains are negative, would therefore change no H-H scores and only 4 of the 213 H-AI scores. We also tested three readouts that, unlike the gap, reward high gains and therefore penalize equally poor gains: the mean gain, the minimum gain, and the gap plus the mean gain (Table R.1). None of them aligns better with human ratings than the current readout, which suggests that how unequal the two gains are tells more about perceived naturalness than how high they are.

ReadoutScoreRewardsH-H Acc. (%)H-H ρ\rhoH-AI ρ\rho
Gap (current)−∣gˉ0−gˉ1∣-\lvert \bar{g}_0-\bar{g}_1 \rvertSimilar gains82.4.281.371
Mean gain(gˉ0+gˉ1)/2(\bar{g}_0+\bar{g}_1)/2High gains of both speakers74.5.262.301
Minimum gainmin⁡(gˉ0,gˉ1)\min(\bar{g}_0,\bar{g}_1)High gain of the lower speaker76.5.278.359
Gap + mean gainz(−∣gˉ0−gˉ1∣)+z(gˉ0+gˉ12)z(-\lvert \bar{g}_0-\bar{g}_1 \rvert)+z\big(\tfrac{\bar{g}_0+\bar{g}_1}{2}\big)Similar and high gains76.5.278.362

Table R.1: Affective readouts on the test set, each fused with the silence component using the paper's weights (.25/.75). Acc. is H-H affective paired accuracy; ρ\rho is Spearman correlation with Combined MOS; z(⋅)z(\cdot) is standardization with development statistics. Bold: best per column.

We therefore keep the current readout.

Revisions.

  • Section 4.3, added after the definition of the affective component: "This gap targets one-sided affective mismatch, in which one speaker behaves naturally while the other's affective state departs from what the context predicts."
  • Appendix F.3: Table R.1 and further readout variants.
  • Appendix F.5 (Limitations), added: "The affective component assumes that at least one speaker behaves naturally. It does not penalize conversations in which both speakers are poorly predicted. Such conversations are rare in CONTACT (both gains below the affect prior in 0 of 100 H-H and 4 of 213 H-AI recordings)."
AuthorsOpenReview ↗

Response to Reviewer 21UN (3/3)

W2: Silence vs. arousal-valence in the affective score

The proposed affective score assigns 75% of its weight to join silence and only 25% to arousal-valence. Removing arousal-valence barely changes the H-H affective MOS correlation (0.28 to 0.27) whereas removing joint silence substantially reduces affective discrimination accuracy (82.4 to 60.8). These results raise concerns that the [scorer] primarily detects temporal disruptions rather than the contextual appropriateness of emotional responses.

Silence is itself an affective signal in conversation, and the two components capture different kinds of affective mismatch.

Thank you for this comment. We agree that joint silence carries much of the H-H affective score. Including silence in the affective score is a deliberate choice. How quickly a partner responds is part of how people perceive emotional engagement. For example, listeners perceive more social connection when response gaps are shorter and less when they are longer [2]. A fitting emotional response that arrives after a long silence can therefore still feel disengaged, which is why the affective score combines vocal affect with joint silence (explained in Section 5.2 in the manuscript). The small H-H change when A-V is removed (.28 to .27) does not mean that A-V carries little signal. Listeners penalize Under-Response much more than Overreaction (MOS 2.30 vs. 3.24; Table 3 in the manuscript), so Under-Response dominates the graded H-H ratings, and it is mainly reflected in silence (Table R.2 below). A-V contributes where silence is weaker: Overreaction in the H-H conversations and the spontaneous mismatches in the H-AI conversations.

The condition-level discrimination results show that the two components are sensitive to different disruptions.

ConditionA-V onlySilence onlyFused
Overreaction76.5 / .74470.6 / .64782.4 / .706
Under-Response47.1 / .58182.4 / .79994.1 / .810
Valence Mismatch58.8 / .59970.6 / .68970.6 / .657

Table R.2: Condition-level discrimination of the affective components on the H-H test set (paired accuracy % / C-index; 17 test sessions, one natural-manipulated pair per session and condition).

A-V is more sensitive to Overreaction and silence to Under-Response, on which A-V is near chance, while the fused score is sensitive to both. Valence Mismatch remains the hardest case for A-V, consistent with the known difficulty of estimating valence from speech [3].

In the H-AI conversations, where mismatches arise spontaneously rather than one kind at a time, removing A-V significantly lowers the affective correlation with human ratings (.371→.319.371 \rightarrow .319; Δρ=.052\Delta\rho=.052, 95% CI [.016,.097][.016, .097]).

The 75% weight on silence was selected on development data to balance the H-H and H-AI settings and then kept fixed. We re-evaluated the same checkpoint with other calibration settings (Table R.3). With equal weights, A-V receives 50%, and the H-AI affective correlation rises from .37 to .42, while the H-H affective correlation falls from .28 to .22. A-V therefore carries useful signal, especially in the H-AI conversations.

SettingNormalizationFusion weightsH-H TTH-H Aff.H-AI TTH-AI Aff.H-AI Overall
(a) PaperCONTACT-devDev-selected.36.28.40.37.36
(b) Equal weightsCONTACT-devEqual.35.22.40.42.37
(c) No CONTACT calibrationSeamless onlyEqual.39.22.40.44.39
Strongest baseline--.19.17.35.26.26

Table R.3: Test Combined-MOS Spearman ρ\rho of the same checkpoint under different calibration settings. Bold: best per column; H-AI TT is .40 in all settings.

Revision. Section 5.2, added: "Including joint silence in the affective score is deliberate, since response timing shapes perceived emotional engagement (Templeton et al., 2022). The two components are sensitive to different disruptions (new Table 5). A-V is more sensitive to Overreaction, silence to Under-Response, and the fused score to both."

We thank the reviewer again; the counterexample in W1.2 helped us describe the score more precisely. We hope our responses address your concerns, and we would be happy to discuss any remaining questions.

[2] E. M. Templeton et al. Fast response times signal social connection in conversation. Proceedings of the National Academy of Sciences, 119(4), 2022.

[3] J. Wagner et al. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 2023.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 21UN,

As the discussion period draws to a close, we wanted to follow up and see whether our responses have addressed your concerns.

We have clarified that the relationship between predictability and perceived naturalness is an empirical hypothesis rather than an assumption. We also evaluated alternative affective readouts (Table R.1), examined the failure case where both speakers are poorly predicted, and provided additional evidence that arousal-valence and silence capture complementary aspects of affective mismatch (new Table 5). In particular, removing the arousal-valence component significantly reduces correlation with human affective ratings in H-AI conversations.

We appreciate your thoughtful comments, which helped us clarify the scorer's formulation and limitations. Please let us know if any concerns remain, and thank you again for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their thoughtful comments. We are encouraged that reviewers found that CONTACT "useful and fills a critical gap" for benchmarking GenAI outputs (pH2S), that it is "a substantive evaluation resource for distinguishing sensitivity to disruptions from alignment with perceived conversational quality" (PkZt), and that it "provides a potentially useful resource for evaluating conversational naturalness" (21UN).

Below we address the two concerns shared across reviews, then the main individual ones; details are in the individual responses. All new analyses reuse the existing recordings and the submitted checkpoint, except one retraining ablation requested by Reviewer PkZt. Main-text changes are highlighted in blue in the revised manuscript.

Data samples. Reviewer pH2S asked for data samples, so we want to make sure every reviewer can access them. Anonymized samples have been available since submission through the repository linked in footnote 1 on page 2 of the manuscript. Its sample gallery (https://review-anonymous-git.github.io/contact-samples/) plays 18 example recordings from all three subsets, each with two-channel audio, video, the conversation prompt, and the participant and supervisor ratings.

Shared concerns

  1. The ratings reproduce across independent listeners (pH2S W3; PkZt Q2). The submission included an independent evaluation of 72 conversations. New independent evaluators, blind to condition, have now rated 212 conversations, including these 72, and wrote a reason for every rating. On the 72, three-rater agreement is higher with the new ratings on every dimension, and across all 212 it is in the same range as participant-supervisor agreement on the full corpus (ICC .51 to .60; Table R.5).

  2. The scorer is not tuned to CONTACT (pH2S W1; PkZt W2, Q3, Q4). No training stage uses CONTACT (Table R.8). The paper calibrates the scores on the CONTACT development split; removing CONTACT from the scoring entirely leaves results the same or better on four of the five dimensions, and the scorer still significantly outperforms the strongest baseline on H-H turn-taking, H-AI affective, and H-AI overall (Table R.3). Removing either refinement objective lowers alignment with human ratings (Table R.9).

Main individual concerns

  1. The elicited behaviors are present in the recordings (PkZt W1, Q1). Every kept conversation passed two recording-time checks: the supervisor confirmed the behavior, and the uninstructed partner, asked without any hint, noticed a difference (Section 3.2). Against the natural conversation of the same session, automatic acoustic measures shift in the instructed direction for all seven conditions they apply to, the partner's rating drops by 1.0 to 2.0 points, and blind evaluators' notes mention the targeted disruption in 51% of manipulated vs. 9% of natural H-H conversations (coded blind to condition by an LLM; Table R.7).

  2. The conditions are well motivated (pH2S W2, Q4). Each reproduces a failure reported for current voice agents, and together they cover the main steps of a turn exchange and errors of both arousal and valence (Table R.4).

  3. Silence in the affective score (21UN W2). Response timing is part of perceived emotional engagement, and the two affective components catch different mismatches (new Table 5); removing arousal-valence significantly lowers the H-AI affective correlation.

  4. The H-AI correlation of .36 (PkZt W2). Our scorer reaches .41 with a single rater, about three quarters of the participant-supervisor agreement (.54), and .50 vs. .35 for the strongest baseline after removing each participant's rating leniency (Appendix F.4).

Summary. CONTACT is, to our knowledge, the first public resource with human naturalness ratings of whole conversations, covering both elicited H-H disruptions and spontaneous H-AI conversations. Our scorer, trained only on natural conversation, aligns with these ratings better than every baseline on all five dimensions.

We hope these additions address the main concerns, and we would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.