- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DyaFDB, a benchmark that evaluates two interacting full-duplex dialogue models across role adherence, information pooling, channel competition, and conflicting goals. Its main study comprises 140 scenarios and 7,560 conversations among three models, supplemented by a fourth-model extension, and analyzes partner dependence, conversational timing, and action versus conditional success.
Critical strengths
-
Treating both speakers as evaluated participants reveals partner-dependent behaviors that a single aggregate score can obscure.
-
The study combines self-play and cross-play with counterbalanced roles, speaking order, and repeated trials.
-
Separating failure to act from failure after acting provides actionable insights beyond task success rates.
Critical weaknesses
-
There is no human-interaction study establishing whether the measured behaviors predict effective, natural conversation; channel dominance and goal attainment alone provide incomplete evidence.
-
The text comparison changes models, prompting, interaction format, and budgets, so performance gaps cannot be attributed specifically to full-duplex dynamics; model-specific adapters introduce additional timing differences.
-
Semantic scoring uses model-generated text rather than the audio heard by the partner, human validation covers only T1, and re-recording garbled sessions may bias estimates of operational performance.
-
Model and partner effects are descriptive and pool-dependent, while symmetric team scores and complementary occupancy scores structurally constrain the decomposition; stronger attribution requires accounting for these constraints and uncertainty.
Questions for authors
-
Can you compare live dyads against matched replay, fixed-examiner, and turn-based conditions to establish which findings specifically require reciprocal full-duplex interaction?
-
How sensitive are the results to MiniCPM-o’s chunking and buffering, model-specific prompt wrappers, and communication-budget matching in the text baselines?
-
What happens when garbled sessions count as observed failures, and do audio-based scoring and human judgments on T2–T4 preserve the reported conclusions?
-
Can you report scenario-level bootstrap confidence intervals for pairing differences and explain how score symmetry, dyad dependence, and model-pool composition affect the variance decomposition?
4 comments
Author Response to Reviewer KPbR (2/2)
[W3 & Q3] Audio scoring, human validation, and garbled sessions
Text-stream scoring or re-recording does not change the conclusions.
Audio/text scoring. We re-judged 780 runs (300 sampled in T1 + 480 sampled in others) from ASR and found it similar to the text scoring, since these full-duplex models are originally designed to output same text and audio semantics. Agreement with the human majority on T1 is 0.89 on ASR and 0.90 on text, and on the judged T2-T4, ASR and text agree on 91% of runs (text re-judgment agrees at 96%). All the model orderings hold except two where text scoring re-judgment also changed ordering.
Human judgments on T2-T4. We validated T1 with human judges because this task needs the most judgment for role adhering. T3's channel occupancy and T4-2's leak are mechanical, and all the other judged labels are closed questions, such as which option was chosen or whether there was a request, which a second judge mostly agrees on (Table 8). Extending the human study to T2-T4 at this size would take ~30 hours of listening, which is impractical.
Garbled sessions. A garbled session is not a valid measurement of either model, because the partner responds to the broken speech, so we re-record it rather than score it. Even if we count every flagged recording as a failure, Raon loses <0.12 and the other models <0.03 on any task.
[W4 & Q4] Uncertainty, symmetry, and pool dependence of variance decomposition
Uncertainty. We bootstrap over scenarios and keep both sides of each conversation together. Changing only the partner significantly changes a model's score in 31/36 turn-taking comparisons and in 23/34 task comparisons outside T1. For T1, it changes in only 1/18, which also matches Figure 2 where T1 is decided by the model itself. Model rankings are also significant in 18/21 comparisons (95% CIs, corrected for multiple testing). For example, Raon wins 0.29 [0.19, 0.38] more of its preferences against PersonaPlex than in self-play (T4-1), and the partner explains 0.65 [0.62, 0.66] of the TOR_end variance.
Score symmetry. In T2 and T3-1, we acknowledge that scoring forces equal model and partner shares, so Section 3.2 has reported these scores only as properties of the pair. The partner effect rests on measures without this constraint (scored separately for each side), namely 65% of turn-end TOR, 49% of barge-in success, and 47% and 51% of the asymmetric T3-2 and T4-2 scores.
Pool composition. Yes, the shares are relative to the pool, as Limitations (Appendix I) stated. However, in all four 3-model subsets of the 4-model pool (Table 15, revised Appendix H), the model side dominates T1, backchannel, and mid-turn take-over rate (TOR_pause), and the partner side dominates T4-2 and barge-in.
| Larger share | Measures |
|---|---|
| Model in all subsets | T1, TOR_pause, backchannel |
| Partner in all subsets | T4-2, barge-in |
| Partner except w/o MiniCPM-o | T4-1, TOR_end |
| Even in all subsets | T3-2 |
TOR_end changes only without MiniCPM-o, because every model takes the turn far more often after MiniCPM-o (Section 3.2). This is the partner effect itself, since a model's score depends on which partner it meets, which a single fixed partner cannot reveal.
Thank you for the detailed rebuttal. The bootstrap analysis strengthens the evidence for partner effects, and the ASR-based rescoring addresses my concern. I’m raising my rating from 2 to 3
Author Response to Reviewer KPbR's Comment
We sincerely thank the reviewer for the valuable feedback and increasing the rating. The reviewer's comments have considerably improved our paper. Thanks!
Author Response to Reviewer KPbR (1/2)
We thank the reviewer for the detailed and constructive feedback. We respond to each concern below, and changes in the revised paper are marked red.
[W1 & Q1] No human-interaction study
Live partner vs. replay or fixed examiner (Q1). Dyadic evaluation (DyaFDB) involves a responsive partner to measure the outcome of an interaction, while none of the prior benchmarks can fully test this. Benchmarks with a fixed partner or a single turn (e.g., FDB, MTR-DB) cannot measure the model's drift or response against other model's behavior. This dynamics appears only when the partner also plays its own role and reacts by staying in the conversation.
A fixed partner, in contrast, reports only one column of our tables:
Table 3), so who's participating with MiniCPM-o could be the main factor.Figure 2).Replaying or fixed examiner smoothes out these variances.
Human partners (W1). As the reviewer has pointed, human partner would keep this interaction live and more realistic, but is not scalable. A single participant would need 84 hours of recording (without self-play) over thousands of conversations. Moreover, since the scores vary by partners, variation across people would be needed to measure. This is why the field recently uses responsive evaluators like models or simulators (e.g., FDB-v2, τ-Voice; see
Table 1).We do not claim that our evaluation protocol predicts effective or natural conversation with people. We evaluate how these full-duplex behaviors vary across interactive partners. We state this in the revised Limitations (Appendix I).
[W2 & Q2] Concerns on the setup of text comparison and adapter
Text comparison. Our text-based comparison experiment (Appendix E) keeps everything fixed except the interaction format from modality change.
Adapter and model wrapping.
Therefore, chunking/buffering is part of the model interface, not a variable for sensitivity study.