- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies conversational naturalness in multi-turn spoken dialogue, focusing on turn-taking and affective appropriateness. It introduces CONTACT, a benchmark containing 493 conversations across three subsets: human–human conversations with elicited turn-taking disruptions, human–human conversations with elicited affective mismatches, and open-ended human–AI conversations. The first two include natural baseline conversations alongside the manipulated conditions, while the third examines spontaneous interactions with spoken dialogue systems. Participants and recording supervisors provide the main human ratings, assessing the corresponding interaction dimension in each human–human subset and turn-taking, affective appropriateness, and overall naturalness in human–AI conversations.
The authors also propose a surprisal-based scorer that predicts future joint voice activity, joint silence, and arousal–valence states from preceding dialogue context. The predictor is trained on approximately 1,636 hours of natural human conversation, without human naturalness-label supervision. At evaluation time, activity and silence likelihoods, together with an affective component based on between-speaker differences in predictive gain, are combined into turn-taking and affective scores. Their mean serves as the overall naturalness score. The method requires no matched reference conversation, while score normalization and fusion weights are determined on development data.
Validation examines both whether scores distinguish natural from manipulated conversations and whether they correlate with graded human ratings, using development and test splits reported to be speaker-disjoint. Comparisons include turn-taking predictors, speech-quality and speech-reward models, an affective-entrainment model, and a timing-statistics control. On CONTACT test sets, the proposed scorer obtains higher correlations with combined participant–supervisor ratings than the evaluated baselines in both human–human subsets and across all three human–AI dimensions. Score-component ablations further show contributions from joint-silence scoring to human–human turn-taking-rating alignment and from arousal–valence scoring to human–AI affective-rating alignment.
Critical strengths
- This paper provides a distinct perspective on evaluating conversational naturalness by examining both disruptions and alignment with human judgements. CONTACT combines controlled human-human conversations with open-ended human-AI dialogue to assess temporal and affective coordination. Its live recordings preserve the partner's responses and subsequent exchange, which are crucial to evaluating a disruption's effect. This makes CONTACT a substantive evaluation resource for distinguishing sensitivity to disruptions from alignment with perceived conversational quality.
- The paper also provides a learning method that explicitly incorporates temporal coordination and affective alignment. Predicting future activity, silence, and arousal-valence states from preceding dialogue addresses the context dependence of conversational appropriateness. Automatically extracted targets allow training on approximately 1,636 hours of natural dialogue, while the smaller CONTACT benchmark supports calibration and human evaluation. Higher correlations with combined human ratings than the evaluated baselines provide empirical support for this approach. The method thus offers a way to learn contextual patterns of interaction at scale and assess their relationship with perceived naturalness, without requiring naturalness ratings throughout the training corpus.
Critical weaknesses
- The benchmark's diagnostic conclusions depend on human ratings and elicited behaviours whose reliability is only partly established. The protocol describes instruction and practice, but does not report how consistently the intended behaviours were realized or their frequency and intensity. Differences attributed to disruption type in Section 3.5 may therefore also reflect variation in enactment. The independent audit covers 72 conversations rated by one additional listener (Appendix D.1), providing limited evidence of reproducibility across independent listeners. These gaps limit confidence in CONTACT as a human-grounded reference for comparing naturalness metrics, which is a central contribution of the paper.
- The scorer's validation leaves uncertainty about the source of its gains and their practical significance for assessing conversational naturalness. Although Table 4 reports significant improvements, statistically significant gains on some comparisons, the human-AI overall correlation with combined ratings is modest at 0.36. The score-removal ablations assess readout components rather than the contributions of training objectives (Table 4; Appendix E.6), while the criterion for selecting CONTACT dev fusion weights is unspecified (Appendix E.3). This makes it difficult to distinguish the contributions of predictive learning and benchmark-specific calibration. The evidence therefore supports comparative improvement more clearly than it establishes the reliability of the proposed design as a measure of conversational naturalness.
Questions for authors
- What perceptual checks confirm that CONTACT recordings convey the intended behaviours or affective expressions? Please report the observed frequency and intensity of the disruptions and affective behaviours, and explain how these support the disruption comparisons.
- What scoring criteria distinguish the intermediate ratings of 2–4? How many of the 493 conversations were independently rated by multiple evaluators, and what evidence supports the reliability and representativeness of the evaluated subset?
- How are the training objectives implemented across stages? Please provide a table of trainable modules and active losses, and clarify how the fusion weights were selected.
- What evidence isolates the contributions of the individual training objectives and CONTACT-dev calibration to test performance? In particular, were retraining and calibration ablations conducted, and how should the reported score-component removal results be interpreted as evidence of component-level contributions?
- To what extent do recording-path delay and cross-device alignment affect perceived naturalness and scorer comparisons? Can these effects be distinguished from the intended interaction effects, beyond the separate latency probe?
7 comments
Response to Reviewer PkZt (2/4)
W1 / Q2: Rating scale, coverage, and the independent audit
The independent audit covers 72 conversations rated by one additional listener (Appendix D.1), providing limited evidence of reproducibility across independent listeners.
What scoring criteria distinguish the intermediate ratings of 2-4? How many of the 493 conversations were independently rated by multiple evaluators, and what evidence supports the reliability and representativeness of the evaluated subset?
Every conversation is rated by both the participant and the supervisor, and 212 conversations are now also rated by new independent evaluators blind to condition.
Scale. Raters used a 5-point scale with labeled endpoints (1 = Very Unnatural, 5 = Very Natural). As in standard MOS protocols [1], the rating is a holistic judgment and no criteria are defined for the intermediate points, which avoids steering raters toward particular behaviors. The new evaluators' written reasons show how the middle of the scale is used in practice. For example, a 3 is often given when one aspect is natural and another is not, such as smooth turn-taking with mismatched emotions.
Coverage. All 493 conversations are rated by the participant and the supervisor. In addition, independent evaluators blind to condition and outside data collection rated 212 conversations (127 H-H and 85 H-AI; I2 in Table R.5), including the 72 rated by the original independent evaluator (I1; Appendix D.1). All independent ratings will be released with the corpus.
Reliability. On the same 72 conversations, using the new independent ratings in place of the original ones raises three-rater agreement on every dimension, and the original and new independent ratings agree with each other at ICC .54 to .76 (Table R.5). Across all 212 conversations, three-rater agreement is moderate on every dimension (ICC .51 to .60), in the same range as the agreement between participant and supervisor on the full corpus (ICC .39 to .70; Table 3 in the manuscript).
| Raters | Conversations | H-H TT | H-H Aff. | H-AI TT | H-AI Aff. | H-AI Overall |
|---|---|---|---|---|---|---|
| P-S-I1 (App. D.1) | 72 | .70 (79%) | .52 (50%) | .61 (58%) | .42 (54%) | .57 (58%) |
| P-S-I2 (new) | 72 | .78 (83%) | .67 (54%) | .63 (58%) | .50 (62%) | .59 (54%) |
| P-S-I2 (new) | 212 | .60 (66%) | .51 (47%) | .54 (56%) | .52 (58%) | .54 (60%) |
| I1-I2 (new) | 72 | .54 (75%) | .57 (71%) | .67 (83%) | .58 (83%) | .76 (92%) |
Table R.5: Rating agreement, ICC(A,1) with Within-1 agreement in parentheses. P: participant; S: supervisor; I1: original independent evaluation; I2: new independent evaluation. The 72 audited conversations include 24 per subset.
Representativeness. The audited subset covers all conditions and all six H-AI systems (four conversations each), and its Combined-MOS distribution is close to that of the full set, for example vs. for H-H turn-taking and vs. for H-AI overall.
Revisions. Section 3.4: "Then, an independent evaluator blind to condition re-rated the conversations (Appendix D.1)." -> "Then, independent evaluators blind to condition re-rated a subset of 212 conversations (Appendix D.1)." Appendix D.1 adds Table R.5 and the subset-versus-full comparison.
[1] ITU-T. Recommendation P.800: Methods for subjective determination of transmission quality. 1996.
Response to Reviewer PkZt (3/4)
W2 / Q3 / Q4: Training objectives, fusion weights, and calibration
The score-removal ablations assess readout components rather than the contributions of training objectives (Table 4; Appendix E.6), while the criterion for selecting CONTACT dev fusion weights is unspecified (Appendix E.3).
How are the training objectives implemented across stages? Please provide a table of trainable modules and active losses, and clarify how the fusion weights were selected.
What evidence isolates the contributions of the individual training objectives and CONTACT-dev calibration to test performance? In particular, were retraining and calibration ablations conducted, and how should the reported score-component removal results be interpreted as evidence of component-level contributions?
No training stage uses CONTACT, ablations show that both refinement objectives contribute, and the gains hold without any CONTACT calibration.
Training stages (Q3). Table R.8 lists what is trained in each stage. No stage uses CONTACT data or labels.
| Stage | Trainable | Frozen | Objectives | Steps |
|---|---|---|---|---|
| 1. Warm-start (from public DualTurn) | Backbone, input projections, all branches and heads | Mimi encoder | Future joint voice activity | 50k |
| 2. Affect refinement | Affect pathway and A/V heads | Backbone, timing pathway | Future affective state | 10k |
| 3. Timing + silence refinement | Timing pathway and heads, silence head | Backbone, affect pathway | Future joint voice activity and future joint silence | 4k |
Table R.8: Trainable modules, frozen modules, and active objectives in each training stage.
Fusion weights (Q3). The timing and affective scores each combine two components with one weight. During model development, each weight was selected on development data to maximize the H-AI correlation while keeping H-H discrimination within a tolerance of equal weights, and was then kept fixed. Appendix E.3 now states this, and the calibration ablation below tests equal weights directly.
Training-objective ablations (Q4, new). Stages 2 and 3 each add one objective (Table R.8): affect refinement and the joint-silence loss. Table R.9 removes each in turn.
| Model | H-H TT Acc. | H-H Aff. Acc. | H-AI TT | H-AI Aff. |
|---|---|---|---|---|
| Full model | 86.0 | 82.4 | .40 | .37 |
| w/o joint-silence loss | 84.0 | 60.8 | .33 | .33 |
| w/o affect refinement (stage 2) | 86.0 | 78.4 | .40 | .33 |
Table R.9: Training-objective ablations on the test set (Combined MOS for ). Rows 1-2: stage 3 retrained for 4k steps with and without the joint-silence loss; without it, the model has no silence score. Row 3: the model trained without stage 2. Because stage 2 updates only the affect pathway and stage 3 neither updates nor uses it, this model combines the affect pathway saved before stage 2 with the full model's activity and silence components.
Removing the joint-silence loss lowers all four measures, most strongly H-H affective accuracy (82.4% to 60.8%). The loss does not change the voice-activity component (84.0% in both runs), so its benefit comes entirely from the silence score it trains, which is why row 2 matches the silence-removal row of Table 4 in the manuscript. Removing affect refinement leaves turn-taking unchanged and lowers the H-AI affective correlation from .37 to .33 (paired 95% CI of the difference ); its H-H change is not significant. For the remaining components, score removal does not isolate a training objective, so we now call these score-component ablations in Section 5.2.
Response to Reviewer PkZt (1/4)
We thank the reviewer for the detailed review and the many concrete questions.
W1 / Q1: Were the intended behaviors realized?
Every retained recording passed recording-time checks, and acoustic measures and blind evaluators' notes confirm that the instructed behaviors occurred.
Recording-time checks. For every manipulated conversation, the research supervisor confirmed that the instructed behavior occurred at least four to five times and was clearly noticeable, and the uninstructed partner, asked afterwards without any hint, noticed a difference from the natural conversation. Conversations that failed either check were recorded again.
Blind evaluators' notes. Because recordings were retained only when the partner noticed a difference, we also checked with listeners outside the recording. The new independent evaluators (see General Response), blind to condition, wrote a reason for every rating. These notes mention the targeted disruption, such as an interruption, a long overlap, an unfinished turn, or flat affect, in 51% of the notes on manipulated H-H conversations, whereas notes on natural conversations mention the same disruptions in only 9% of cases.
Per-condition checks. Table R.7 compares each manipulated conversation with the natural conversation of the same session, so the speakers and recording setup are held fixed.
Table R.7: Manipulated minus natural conversation of the same session. Blind notes: notes mentioning the targeted disruption, manipulated vs. natural (coded blind to condition by an LLM with a fixed codebook). Partner: the speaker who received no instruction. *: 95% CI excludes zero; all MOS changes are significant (Wilcoxon p<.001). A retake is the instructed speaker resuming within 1 s after the partner starts. Competitive Floor Conflict instructs both speakers. Definitions, sample sizes, and CIs are in Appendix F.2.
All five turn-taking measures change in the instructed direction, and the instructed speaker's arousal rises in Overreaction and falls in Under-Response. Valence Mismatch has no acoustic entry because it is defined relative to each topic's intended valence, which a recording-level acoustic measure does not capture. It is supported by the recording-time checks, the blind notes, and the largest partner rating drop of any condition.
Revisions. Section 3.2 now describes the recording-time checks. New Appendix F.2 reports Table R.7 with all measures, definitions, and CIs.