acceptodds
Under review as a conference paper at ICLR 2027

Conversation Is a Two-Body Problem: Dyadic Evaluation of Full-Duplex Dialogue Models

Abstract

Full-duplex spoken dialogue models listen and speak at the same time, enabling voice agents to have natural, low-latency interactions that turn-based systems cannot offer. However, they are commonly evaluated against single-sided interlocutors: pre-recorded audio that cannot react, or an automated examiner that reacts in real time but only administers a fixed sequence of tests and is never graded. These single-sided frameworks evaluate only half of a *two-body problem*, where turn-taking, overlap, and interruption are joint products of two coupled speakers. We propose **DyaFDB**, a framework that evaluates full-duplex models in a dyadic setup: two models converse directly under assigned roles with cooperative or conflicting goals, and both sides are scored offline with an external judge. DyaFDB probes how the two models behave toward each other, such as how they take turns or carry an assigned role under different interests. We instantiate four tasks as 140 scenarios and record 7,560 conversations, covering six self- and cross-play pairings. Throughout the experiments, we observe that how a model behaves continually reshapes its partner. We thus demonstrate that each model must be both the examiner and examinee of the other, and no single fixed interlocutor can play both parts. We will release the scenarios, role prompts, and recording protocols between two full-duplex models, without any pre-recorded audio.

Primary area:
datasets and benchmarks
Keywords:
full-duplex dialogue models, dyadic evaluation benchmark, model-to-model interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KPbROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DyaFDB, a benchmark that evaluates two interacting full-duplex dialogue models across role adherence, information pooling, channel competition, and conflicting goals. Its main study comprises 140 scenarios and 7,560 conversations among three models, supplemented by a fourth-model extension, and analyzes partner dependence, conversational timing, and action versus conditional success.

Critical strengths
  • Treating both speakers as evaluated participants reveals partner-dependent behaviors that a single aggregate score can obscure.

  • The study combines self-play and cross-play with counterbalanced roles, speaking order, and repeated trials.

  • Separating failure to act from failure after acting provides actionable insights beyond task success rates.

Critical weaknesses
  • There is no human-interaction study establishing whether the measured behaviors predict effective, natural conversation; channel dominance and goal attainment alone provide incomplete evidence.

  • The text comparison changes models, prompting, interaction format, and budgets, so performance gaps cannot be attributed specifically to full-duplex dynamics; model-specific adapters introduce additional timing differences.

  • Semantic scoring uses model-generated text rather than the audio heard by the partner, human validation covers only T1, and re-recording garbled sessions may bias estimates of operational performance.

  • Model and partner effects are descriptive and pool-dependent, while symmetric team scores and complementary occupancy scores structurally constrain the decomposition; stronger attribution requires accounting for these constraints and uncertainty.

Questions for authors
  • Can you compare live dyads against matched replay, fixed-examiner, and turn-based conditions to establish which findings specifically require reciprocal full-duplex interaction?

  • How sensitive are the results to MiniCPM-o’s chunking and buffering, model-specific prompt wrappers, and communication-budget matching in the text baselines?

  • What happens when garbled sessions count as observed failures, and do audio-based scoring and human judgments on T2–T4 preserve the reported conclusions?

  • Can you report scenario-level bootstrap confidence intervals for pairing differences and explain how score symmetry, dyad dependence, and model-pool composition affect the variance decomposition?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer KPbR (1/2)

We thank the reviewer for the detailed and constructive feedback. We respond to each concern below, and changes in the revised paper are marked red.


[W1 & Q1] No human-interaction study

Live partner vs. replay or fixed examiner (Q1). Dyadic evaluation (DyaFDB) involves a responsive partner to measure the outcome of an interaction, while none of the prior benchmarks can fully test this. Benchmarks with a fixed partner or a single turn (e.g., FDB, MTR-DB) cannot measure the model's drift or response against other model's behavior. This dynamics appears only when the partner also plays its own role and reacts by staying in the conversation.

A fixed partner, in contrast, reports only one column of our tables:

  • MiniCPM-o apparently solves 30% of information pooling tasks with Raon but only 7% with PersonaPlex (Table 3), so who's participating with MiniCPM-o could be the main factor.
  • The partner explains 65% of the turn-end take-over rate (TOR_end) variance (Figure 2).

Replaying or fixed examiner smoothes out these variances.

Human partners (W1). As the reviewer has pointed, human partner would keep this interaction live and more realistic, but is not scalable. A single participant would need 84 hours of recording (without self-play) over thousands of conversations. Moreover, since the scores vary by partners, variation across people would be needed to measure. This is why the field recently uses responsive evaluators like models or simulators (e.g., FDB-v2, τ\tau-Voice; see Table 1).

We do not claim that our evaluation protocol predicts effective or natural conversation with people. We evaluate how these full-duplex behaviors vary across interactive partners. We state this in the revised Limitations (Appendix I).


[W2 & Q2] Concerns on the setup of text comparison and adapter

Text comparison. Our text-based comparison experiment (Appendix E) keeps everything fixed except the interaction format from modality change.

  • Model: The LLM pair includes Qwen3-8B, which is the backbone of both MiniCPM-o and Raon, and Qwen2.5-7B.
  • Prompting: The same scenarios and role prompts, with one paragraph added asking for brief replies.
  • Budget: The one-minute window becomes a 240-word budget, based on our measures of the speech model's talking speed, so the communication budget is already matched.
  • Interaction format: What's different is that the text pair exchanges turns, while the duplex models listen and speak at once in real time (i.e., speaking dynamics).

Adapter and model wrapping.

  • Chunking buffering: MiniCPM-o's chunking does not come from our adapter; its own duplex API reads ~1 sec of audio at a time and decides per chunk whether to speak (Appendix A). The adapter is there only to feed API the frames and return its output, and to make the shared lockstep clock wait while the model computes so that the partner does not go ahead of what MiniCPM-o has heard.
  • Prompt wrapper: The model's own official system prompt for spoken chat, which carries the same role prompt every model gets. Without this, the model just reads the role prompt aloud.

Therefore, chunking/buffering is part of the model interface, not a variable for sensitivity study.

AuthorsOpenReview ↗

Author Response to Reviewer KPbR (2/2)

[W3 & Q3] Audio scoring, human validation, and garbled sessions

Text-stream scoring or re-recording does not change the conclusions.

Audio/text scoring. We re-judged 780 runs (300 sampled in T1 + 480 sampled in others) from ASR and found it similar to the text scoring, since these full-duplex models are originally designed to output same text and audio semantics. Agreement with the human majority on T1 is 0.89 on ASR and 0.90 on text, and on the judged T2-T4, ASR and text agree on 91% of runs (text re-judgment agrees at 96%). All the model orderings hold except two where text scoring re-judgment also changed ordering.

Human judgments on T2-T4. We validated T1 with human judges because this task needs the most judgment for role adhering. T3's channel occupancy and T4-2's leak are mechanical, and all the other judged labels are closed questions, such as which option was chosen or whether there was a request, which a second judge mostly agrees on (Table 8). Extending the human study to T2-T4 at this size would take ~30 hours of listening, which is impractical.

Garbled sessions. A garbled session is not a valid measurement of either model, because the partner responds to the broken speech, so we re-record it rather than score it. Even if we count every flagged recording as a failure, Raon loses <0.12 and the other models <0.03 on any task.


[W4 & Q4] Uncertainty, symmetry, and pool dependence of variance decomposition

Uncertainty. We bootstrap over scenarios and keep both sides of each conversation together. Changing only the partner significantly changes a model's score in 31/36 turn-taking comparisons and in 23/34 task comparisons outside T1. For T1, it changes in only 1/18, which also matches Figure 2 where T1 is decided by the model itself. Model rankings are also significant in 18/21 comparisons (95% CIs, corrected for multiple testing). For example, Raon wins 0.29 [0.19, 0.38] more of its preferences against PersonaPlex than in self-play (T4-1), and the partner explains 0.65 [0.62, 0.66] of the TOR_end variance.

Score symmetry. In T2 and T3-1, we acknowledge that scoring forces equal model and partner shares, so Section 3.2 has reported these scores only as properties of the pair. The partner effect rests on measures without this constraint (scored separately for each side), namely 65% of turn-end TOR, 49% of barge-in success, and 47% and 51% of the asymmetric T3-2 and T4-2 scores.

Pool composition. Yes, the shares are relative to the pool, as Limitations (Appendix I) stated. However, in all four 3-model subsets of the 4-model pool (Table 15, revised Appendix H), the model side dominates T1, backchannel, and mid-turn take-over rate (TOR_pause), and the partner side dominates T4-2 and barge-in.

Larger shareMeasures
Model in all subsetsT1, TOR_pause, backchannel
Partner in all subsetsT4-2, barge-in
Partner except w/o MiniCPM-oT4-1, TOR_end
Even in all subsetsT3-2

TOR_end changes only without MiniCPM-o, because every model takes the turn far more often after MiniCPM-o (Section 3.2). This is the partner effect itself, since a model's score depends on which partner it meets, which a single fixed partner cannot reveal.

Thank you for the detailed rebuttal. The bootstrap analysis strengthens the evidence for partner effects, and the ASR-based rescoring addresses my concern. I’m raising my rating from 2 to 3

AuthorsOpenReview ↗

Author Response to Reviewer KPbR's Comment

We sincerely thank the reviewer for the valuable feedback and increasing the rating. The reviewer's comments have considerably improved our paper. Thanks!

Reviewer_YMHnOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark called DyaFDB, which evaluates full-duplex dialogue models through live interactions between two models rather than against a fixed examiner or prerecorded audio. The main contributions are the dyadic evaluation framework, which is tested over four tasks that cover role adherence, information pooling, competition for the speaking channel and conflicting goals, instantiated in 140 scenarios.

The authors contribute 7560 conversations across self-play and cross-play pairings of three open-weight models, showing that conversational behavior depends strongly on the partner, which is an important result for full-duplex evaluations in general. For example, the experiments reveal substantial partner-dependent variation in turn-taking and task outcomes. The partner explains 65% of the variance in turn-end takeover, and secret-extraction success varies considerably across attackers. Overall their findings highlight the limitations of evaluating full-duplex models against a single fixed interlocutor.

Critical strengths
  1. The paper provides convincing evidence that full-duplex conversational behavior depends substantially on the interaction partner. Figure 2 for example shows that 74% of pause-takeover variance is attributed to the evaluated model, whereas 65% of turn-end takeover variance is attributed to its partner. Backchanneling also exhibits a substantial model–partner interaction effect (32%). These contrasting results validate the paper's central argument that different conversational behaviors depend on different participants, making fixed-interlocutor evaluations potentially misleading.

  2. The experiments identify distinct failure mechanisms rather than merely reporting task success. For example, PersonaPlex's role adherence drops from 0.51 to 0.29 with a digressing partner, while Raon's increases from 0.68 to 0.83 (Table 3). More importantly, Figure 4 separates failures to act from failures after acting, so MiniCPM-o reaches a decision in only 41% of information-pooling conversations, despite achieving 40% accuracy when it does decide. This distinction helps identify whether poor performance stems from insufficient engagement or unsuccessful task execution.

  3. The evaluation is extensive over 126 hours of audio covering self-play and cross-play with counterbalanced roles and speaking order. They also examine sensitivity to conversation duration (30–120 seconds), validate LLM-based scoring against human judgments. These analyses strengthen confidence in the reported findings.

Critical weaknesses
  1. The benchmark measures turn-taking and task outcomes without assessing whether conversations are coherent or contextually appropriate. For example, T3 measures channel dominance through exclusive speaking time, where Raon achieves 0.75 occupancy against MiniCPM-o (Table 4), without distinguishing productive engagement from excessive speech. Similarly, role adherence does not necessarily imply meaningful interaction. This limits the benchmark's ability to assess conversational competence beyond observable behaviors.

  2. The paper's central conclusions rely heavily on variance decomposition across just three models. However, these percentages are relative to the specific models included, rather than intrinsic properties of conversational behaviors. Appendix H shows this when adding Lychee-FD changes the secret-extraction variance decomposition such that 81% is attributed to the defender, largely because Lychee-FD is unusually resistant to extraction. This raises concerns about how generalizable the reported model-versus-partner effects are, especially since the paper uses these estimates to motivate its evaluation framework.

Questions for authors
  1. Have the authors examined sensitivity to alternative prompt formulations?

  2. The partner hears generated audio, but the LLM judge evaluates internal text streams. Appendix A reports audio–text inconsistencies in approximately 5% of Raon sessions, detected using Whisper and handled through re-recording. Have the authors considered using ASR transcripts of the actual generated audio for scoring instead?

  3. In the information-pooling task (T2), the spoken model pairs solve only 5–35% of scenarios (Table 3), whereas the text-based Qwen3-8B comparison in Appendix E achieves 55%. How much of this gap arises from failures to accurately perceive the partner's speech versus failures to communicate the private information or reason over it?

  4. Section 5.2 shows that extending conversations from 60 to 120 seconds increases information disclosure in T2 but does not meaningfully improve task success. Is this due to models failing to integrate the exchanged information, losing track of earlier statements, or failing to reach a final decision (or some other failure mode)?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer YMHn

We thank the reviewer for the detailed and constructive feedback. We respond to each concern below, and changes in the revised paper are marked red.


[W1] Conversation appropriateness is not assessed

We agree with the reviewer that evaluating conversational quality is an important problem. However, measuring coherence and appropriateness first requires a reliable way to evaluate full-duplex models in interaction. Prior frameworks set the partner fixed, so what they report depended on the partner they chose. As a pioneering step, DyaFDB proposes such a novel framework and measures behavior rather than quality, as we stated in Limitations (Appendix I).

Similarly, T3-1 measures only who holds the channel well when both sides compete for it, so Raon's 0.75 is the outcome of that contest, not a claim that it converses well. T3-2 measures whether the presser actually voiced its request and whether the partner granted it (Figure 4), and T1 judges whether the model behaved as its assigned role throughout the conversation. Quality measures can be built on this framework, which we leave to future work.


[W2] Variance decomposition depends on the three-model pool

We agree that the variance shares are relative to the pool, as stated in Limitations (Appendix I), but our main claim that the partner's behavior is a strong factor still holds. Recalculated with all four 3-model subsets of the 4-model pool (Table 15, revised Appendix H), the model side dominates T1, backchannel, and mid-turn take-over rate (TOR_pause), and the partner side dominates T4-2 and barge-in.

Larger shareMeasures
Model in all subsetsT1, TOR_pause, backchannel
Partner in all subsetsT4-2, barge-in
Partner except w/o MiniCPM-oT4-1, TOR_end
Even in all subsetsT3-2

Turn-end take-over rate (TOR_end) changes only without MiniCPM-o, because every model takes the turn far more often after MiniCPM-o (Section 3.2). This is the partner effect itself, since a model's score depends on which partner it meets, which a single fixed partner cannot reveal.

Adding Lychee-FD raised the partner effect of T4-2 (defender) to 0.81 because a resistant defender increased variations, but still the partner effect is large. While variance decomposition is a quantitative metric to see the overall trend, its effect is also visible in the scores without any decomposition. For instance, Raon defender leaks the secret in 39% of conversations against itself but 63% against PersonaPlex (Table 5).


[Q1] Sensitivity to prompting

Each score we reported averages over 20 scenarios that vary the setting, such as personas and speaking manners, although we did not test paraphrases of the same prompts. For instance, we have a partner who is "unhurried and playful with words", or whose "answers stay clipped and small". Model rankings also hold when we resampled scenarios. We plan to release all prompts for a comprehensive study.


[Q2] Scoring with ASR transcripts

We re-judged 780 runs (300 sampled in T1 + 480 sampled in others) from ASR and found it similar to the text scoring, since these full-duplex models are originally designed to output same text and audio semantics. The inconsistency arises only in Raon's speech decoding, which our ASR catches and re-records (Appendix A, updated with the complete re-recording stats). Agreement with the human majority on T1 is 0.89 on ASR and 0.90 on text, and on the judged T2-T4, ASR and text agree on 91% of runs (text re-judgment agrees at 96%). All the model orderings hold except two where text scoring re-judgment also changed ordering.


[Q3] Source of the speech-text gap in T2

Both the communication and using what's heard contribute.

  1. Communicating the private information: The speech models disclose fewer of their private information (at most 47%, compared to 74% for Qwen3-8B in the text pair), so less information reaches to their partner.

  2. Reasoning over it: Yet even in runs where both sides disclosed, speech pairs that reach a decision are correct only 44% of the time, compared to 72% for the text pair.

Even with the information on the table, the speech pairs struggle to turn what they've heard into a correct joint decision with their partner.


[Q4] Why 120s does not raise T2 success

It's not a decision failure. From 60s to 120s, pairs decide +8.1% more often but solve only +0.6% more, so almost all the added decisions are wrong. The largest reason (4.2%) is that they pick a candidate whose excluding reason had already been said in the conversation. This is the same bottleneck as in Q3 above, since more information does not lead these models to the right decision. The failure thus lies in integrating or keeping track of what was exchanged, not in reaching a decision.

Reviewer_9H4sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DyaFDB, which is based on the core idea that evaluating full-duplex dialogue models should not rely solely on pre-recorded speech or fixed human testers. Instead, such models should engage in genuine, real-time conversations with one another, with the performance of both participants being evaluated simultaneously. The authors have designed four categories of tasks encompassing seven sub-task types, covering scenarios such as role maintenance, information sharing, competition for the floor, and goal conflict.

Critical strengths

1.The task design is comprehensive. It encompasses both collaborative and competitive scenarios, evaluating not only semantic-level coordination and decision-making but also speech-level dynamics such as interjections, interruptions, and turn-taking control. This offers a richer assessment than benchmarks that focus solely on turn-taking.

Critical weaknesses
  1. Lack of validation through authentic human-machine interaction. Although the paper emphasizes that dual-model interaction more closely resembles real-world conversation, the actual experiments rely exclusively on model-to-model dialogue. Interactions between models may develop specific behavioral patterns that differ from the pauses, interruptions, responses, and collaborative behaviors characteristic of human interaction.
  2. Limited validity of conversational metrics Current models exhibit limited spontaneous initiation and inconsistent conversational behaviors. Moreover, metrics such as channel occupancy, backchannel frequency, and barge-in success do not necessarily reflect conversational appropriateness or naturalness. 3.Limited evaluation of strong full-duplex systems The experiments are restricted primarily to three open-weight models, excluding stronger commercial systems such as Gemini-Live and GPT-Live. Without evaluating more capable conversational agents, it is unclear whether the findings characterize general dyadic interaction or merely reflect limitations of the selected models. 4.Insufficient differentiation from adaptive simulation benchmarks Existing benchmarks such as Tau-Voice already support dynamic full-duplex audio interaction. The paper does not establish what additional evaluation validity is gained by replacing an adaptive user simulator with another evaluated full-duplex model.
Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 9H4s

We thank the reviewer for the detailed and constructive feedback. We respond to each concern below, and changes in the revised paper are marked red.


[W1] No human-interaction study

Involving human partner would keep the interaction live and more realistic, but is not scalable. A single participant would need 84 hours of recording (without self-play) over thousands of conversations. Moreover, since the scores vary by partners, variation across people would be needed to measure. This is why the field recently uses responsive evaluators like models or simulators (e.g., FDB-v2, τ\tau-Voice; see Table 1). DyaFDB instead evaluates the speaking dynamics of both full-duplex participants, which none of the prior benchmarks could test, across wide range of scenarios, roles, and counterbalanced openers in 7560 conversations.

We do not claim that our evaluation protocol predicts effective or natural conversation with people. We evaluate how these full-duplex behaviors vary across interactive partners. In the revision, we made two changes:

  • Conclusion: we replaced "in a realistic interaction setup" with "in a live interaction with a responsive partner".
  • Limitations (Appendix I): we added that every partner in this study is a model and placing real human partners is a future work.

[W2] Conversation appropriateness is not assessed

We agree with the reviewer that evaluating conversational quality is an important problem. However, measuring appropriateness or naturalness first requires a reliable way to evaluate full-duplex models in interaction. Prior frameworks set the partner fixed, so what they report depended on the partner they chose. As a pioneering step, DyaFDB proposes such a novel framework and measures behavior rather than quality, as we stated in Limitations (Appendix I).

Channel occupancy, backchannel frequency, and barge-in success all describe how these pairs take turns, not whether it does so appropriately. Also, we have discussed the limited spontaneous initiation and inconsistent behavior of these models in Limitations (Appendix I). Figure 4 thus separated whether a model initiated actions from whether it succeeded after acting, and we demonstrated in Section 5.1 that a low task score often comes from a model that never acted. Quality measures can be built on this framework, which we leave to future work.


[W3] No commercial systems tested

Our scope of this study is to evaluate open-weight full-duplex models that support prompt-based conversations at a lockstep clock via frame exchange (Section 3.1) so that anyone can rerun the benchmark on the same fixed checkpoints. Extending this framework to closed systems like GPT-Live could be a next step, which requires bridging their real-time streaming APIs with network latency to the shared clock.

The results being relative to the model pool are a normal property of pool-based evaluation like Chatbot Arena, as we have stated in Limitations (Appendix I). We demonstrated that adding a fourth model, Lychee-FD, keeps the overall trend, and when recalculated with all four 3-model subsets of the 4-model pool (Table 15, revised Appendix H), the model side dominates T1, backchannel, and mid-turn take-over rate (TOR_pause), and the partner side dominates T4-2 and barge-in.

Larger shareMeasures
Model in all subsetsT1, TOR_pause, backchannel
Partner in all subsetsT4-2, barge-in
Partner except w/o MiniCPM-oT4-1, TOR_end
Even in all subsetsT3-2

Turn-end take-over rate (TOR_end) changes only without MiniCPM-o, because every model takes the turn far more often after MiniCPM-o (Section 3.2). This is the partner effect itself, since a model's score depends on which partner it meets, which a single fixed partner cannot reveal. Thus, our main claim that the partner's behavior is a strong factor still holds.


[W4] Missing comparison with τ\tau-Voice

We thank the reviewer for pointing to the τ\tau-Voice benchmark, which we added to Table 1, Related Work, and Appendix E in the revision. We highlight that the two benchmarks measure different things and complement each other.

In τ\tau-Voice, the user is a text LLM that receives the voice agent's transcript and speaks through TTS, so it never hears the agent's audio dynamically. Moreover, it yields after a configurable overlap duration, which is their fixed setting. In DyaFDB, that reaction is what we measure, since the partner is itself an evaluated full-duplex model that listens and decides when to yield.

In turn, τ\tau-Voice evaluates what DyaFDB does not, whether an agent completes verifiable customer-service tasks with tool-calling under realistic background noise and diverse accents.

Thank you for the detailed response and additional experiments. I appreciate the clarifications, though I still have some concerns about how well interactions between current full-duplex models reflect meaningful conversational capabilities. I would also be interested in seeing evaluations with stronger systems and a clearer comparison with adaptive simulators such as τ-Voice to better understand the unique value of the proposed setting. Overall, I appreciate the contribution but will maintain my current rating.

AuthorsOpenReview ↗

Author Response to Reviewer 9H4s's Comment

We thank the reviewer about the valuable feedback. We agree that evaluation with strong commercial systems is a truly interesting next step by bridging real-time APIs. The comparison with a simulator (like tau-voice) would be another approach to confirm the value of our dyadic evaluation, but we believe that the evidence and findings in our paper clearly support the DyaFDB's evaluation framework. Still we appreciate that the reviewer's feedback will help advance benchmarking conventions in full-duplex speech models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.