acceptodds
Under review as a conference paper at ICLR 2027

DuplexSpeechBench–IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

Abstract

Full-duplex voice agents must continuously decide when to speak, listen, backchannel, interrupt, overlap, and yield the conversational floor. Existing benchmarks evaluate these behaviors through explicit turn-management instructions, whereas voice agents are often configured through roles or personas from which appropriate conversational behavior must be inferred. We introduce DuplexSpeechBench–IFEval (DSB-IFEval), a benchmark for evaluating implicit instruction following in real-time spoken interaction. DSB-IFEval comprises 1,038 test cases derived from 240 controlled conversations spanning eight behaviorally contrastive assistant roles and five conditioning protocols: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona–rule conditioning, and instruction conflict. We measure real-time floor management using the deterministic Instruction Adherence Score (IAS) and persona-consistent response content using the LLM-judged Persona Adherence Score (PAS). Across eleven real-time speech models, we find that executing an explicit floor-management policy does not reliably imply the ability to infer the same policy from a persona. Moreover, even frontier models such as GPT-Live-1 and Gemini-3.8-Live adapt their dialogue language to the assigned persona without consistently translating that persona into the appropriate full-duplex floor-management behavior. Finally, models that successfully resolve benign instruction conflicts often fail when safety-relevant role behavior should override an explicit directive. These results show that inferring role-implied behavior, executing it in real time, and resolving instruction conflicts remain distinct challenges for full-duplex voice agents.

Primary area:
datasets and benchmarks
Keywords:
ull-duplex speech, voice agents, instruction following, implicit instruction following, persona conditioning, spoken dialogue, turn-taking, speech language models, real-time interaction, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_rhXPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper puts an additional evaluation aspect on implicit instruction following by defining role-grounded full-duplex behaviors. It compares different instruction and persona conditions, and measures the performance through timing- and persona-related metrics.

Critical strengths

Role-grounded full-duplex behaviors are still underexplored and worth being highlighted.

Critical weaknesses

Writing: I believe this work needs significant improvement in its writing to meet the ICLR bar.

  • Table 1 is never mentioned in the main text. The Introduction should further clarify the positioning of the proposed work in plain language.

  • Clearly mention that DSB-IFEval is the abbreviated form of the full name.

  • Lines 84 and 304 have missing citations.

  • Is “conversational probe” a widely used term in the field? This does not align with my initial understanding of “probing.” I want the authors to clarify whether the term was carefully selected by them rather than generated by AI.

  • There is no justification for shared/specific probes, and I could not find a clear difference between them. For example, the examples in lines 215–216 do not clearly support role-specific behaviors.

  • The descriptions of L1–L4 are too abstract to understand clearly. A concrete example would help readers.

  • Important information is included only in appendices that are insufficiently connected to the main text. For example, Table 30 shows that the experimental setups for L0 and L4b differ from those of the other conditions. Reviewers should not have to discover this information accidentally. The authors should reorganize the manuscript and address these details in the main text.

Motivation

  • It is not very clear why the authors position DSB-IFEval as implicit instruction following, as it addresses explicit instructions with persona behaviors. This could be called “instruction following with a persona,” but not “implicit instruction following.”

Resource Novelty

  • If the proposed benchmark is intended for future use, the implicit behaviors associated with each persona should have a generalizable basis. However, all behaviors are arbitrarily defined without any evidence. The current setup, which defines the behaviors first and then asks humans whether they agree, is not a grounded approach. Previous works, such as https://arxiv.org/abs/2506.21463 and https://arxiv.org/abs/2607.26178, have already attempted to align full-duplex behaviors with human preferences. The authors also do not provide evidence that the listed role behaviors are sufficiently diverse. For a reusable benchmark, sufficient justification is required.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Official Review Response

We thank the reviewer for the detailed feedback and for recognizing that role-grounded full-duplex behavior is an underexplored research problem. We appreciate the valuable feedback and address each concern below.

1. Writing, presentation, and positioning of the benchmark

We thank the reviewer for providing valuable feedback to improve the presentation of the paper. We will make the following presentation improvements:

  • Introduction and Table 1: Explicitly reference Table 1 in Sec. 1 and explain how DSB-IFEval differs from benchmarks evaluating conversational dynamics or explicit turn-taking instructions.
  • Abbreviation: Clearly introduce DuplexSpeechBench–IFEval (DSB-IFEval) at first mention.
  • Missing citations: Correct the missing Fun-Audio-Chat references at Lines 84 and 304 using the Fun-Audio-Chat Technical Report (Chen et al., 2025; arXiv:2512.20156).
  • Methodology: Move representative role, scenario, and conditioning examples from Appendix B into Sec. 3 to make the experimental setup understandable without consulting the appendix.
  • Appendix references: Add direct references to the corresponding validation, implementation, and case-composition tables when introducing the relevant methods.

2. Use of the term "conversational probe" and distinction between shared and role-specific probes

We use the term probe in the diagnostic sense of a controlled conversational event designed to elicit a measurable behavioral decision. We do not intend to claim that "conversational probe" is an established term or a new probing methodology. As authors, we take responsibility for the choice and use of this terminology. To avoid confusion, we will replace it with controlled conversational test scenarios where appropriate in Sec. 3.2.

3. Justification for shared and role-specific scenarios

The two scenario types serve distinct experimental purposes:

  • Shared scenarios (C1–C4): Non-terminal pause, structured dictation, user barge-in, and attention check. These allow comparison of role-dependent behavior under analogous conversational events.
  • Role-specific scenarios (two per role): Target distinctive role obligations, such as intervening on a clinical red flag, preserving extended meditation silence, or correcting an incorrect mathematical reasoning step.

For example, Appendix B.3 (Table 16) contains:

RoleScenarioExpected behavior
ER triage nurseUrgent clinical red flagINTERRUPT
Meditation instructor11-second silenceLISTEN
Socratic math tutorIncorrect reasoning stepINTERRUPT
Improv scene partnerIn-character overlapping speechACCEPT OVERLAP

Thus, the scenarios test different behavioral requirements, not merely different dialogue topics. We will add this distinction and a representative table to Sec. 3.2, while retaining the complete specifications in Appendix B.3.

4. Concrete explanation of the L0–L4 protocols

We agree that an example will make the conditioning levels easier to understand. We will move the grief-counselor example from Appendix B.5 into Sec. 3.1 and revise Figure 2 accordingly.

Consider a grief counselor interacting with a user who pauses during a difficult disclosure:

ConditionModel receives
L0No persona or behavioral instruction
L1Explicit instruction to preserve a reflective pause
L2Grief-counselor persona, without the explicit silence rule
L3Persona together with the explicit silence rule
L4aPersona with a conflicting instruction to fill silence

L4b separately tests safety-critical conflicts in which a role-implied obligation should override an explicit directive. The comparison therefore evaluates whether a model can infer and execute the same policy when it is communicated implicitly rather than explicitly.

AuthorsOpenReview ↗

Official Review Response Continued

5. Differences in experimental coverage across conditioning levels

These differences are intentional, but we agree they should be visible in the main paper. The benchmark contains 240 unique conversations (8 roles × 6 scenarios × 5 instances), expanded into 1,038 evaluation cases:

ConditionsCases per conditionPurpose
L048Default behavior, one instance per role–scenario cell
L1, L2, L3, L4a240 eachFull matched evaluation grid
L4b30Six safety-eligible cells × five instances

Thus, L1–L3 support the central matched explicit-versus-implicit comparisons, whereas L0 is a smaller baseline and L4b a targeted safety-conflict evaluation. We will move these counts into Sec. 3.2, update Table 3's caption, and explicitly document the scoring denominators and paired comparison sets, with the complete selection rules remaining in Appendix G.4 (Table 30).

6. Why DSB-IFEval evaluates implicit instruction following

We respectfully disagree that our benchmark is simply "instruction following with a persona." The key distinction is whether the behavioral policy is explicitly specified or must be inferred.

  • L1 (Explicit): The model receives a direct instruction specifying the required floor-management behavior.
  • L2 (Implicit): The model receives only a persona description. It must infer the appropriate behavior from the role and conversational context, then execute it in real time.
  • L3 (Combined): The implied rule is restated explicitly, allowing us to examine whether additional behavioral specification improves execution.

Crucially, the user-side audio and timing are held fixed across these conditions. This controlled comparison distinguishes the challenge of inferring a policy from executing an explicitly provided one. We also prevent personas from trivially restating the target instructions through imperative-leakage and lexical-overlap checks (Appendix B.6). Our results demonstrate why this distinction matters. For example, Gemini-3.8-Live's PAS increases from 32.8 to 66.8 under persona conditioning (L0→L2), whereas IAS remains approximately unchanged (29.0→28.6). Similarly, StepAudio-3-Realtime's PAS increases from 29.0 to 68.7, without improved IAS (35.5→33.5). These findings expose a meaningful gap between adopting persona-consistent language and translating a persona into appropriate real-time floor behavior. We will make this definition explicit in Secs. 1 and 3.1 and emphasize L2 as the primary implicit-instruction condition, with L1/L3/L4 serving as diagnostic comparisons.

7. Human grounding of persona-to-behavior mappings

We respectfully disagree that the role behaviors are arbitrarily defined without independent evidence. The submitted paper already includes a blinded human-validation study specifically addressing this concern (Appendix A.1, H1). Annotators received the L2 persona and user scenario but were not shown the explicit L1 instruction or the benchmark's expected action. Importantly, annotators were not asked to agree with an author-provided rule; they inferred the behavior independently without seeing the intended answer. This directly addresses the concern that the validation merely confirms predefined mappings. We further acknowledge that independently recoverable behavior is not necessarily the universally preferred behavior in every real-world interaction. We will explicitly distinguish these concepts and discuss less-unambiguous roles, including the 911 dispatcher and simultaneous interpreter. We will bring the human-study methodology and key findings into Sec. 3.2, rather than leaving this evidence solely in Appendix A.1.

AuthorsOpenReview ↗

Official Review Response Continued

8. Behavioral diversity and relationship to prior research

Our eight roles were selected for behavioral contrast rather than occupational diversity alone. Table 2 defines their policies along five dimensions: interruption, backchanneling, silence tolerance, readback, and overlap handling. For example, prolonged silence may be appropriate for a meditation instructor but not for a dispatcher resolving urgent ambiguity. Similarly, an interpreter may continue through brief overlap, whereas a triage nurse may need to yield immediately. We will strengthen Sec. 3.2 with an explicit role-by-behavior comparison demonstrating these contrasts and connecting it to the matched-audio diagnostic in Table 6, where byte-identical user speech is evaluated under three role-specific behavioral targets.

We also appreciate the suggested references:

  • Aligning Spoken Dialogue Models from User Interactions (arXiv:2506.21463).
  • DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues (arXiv:2607.26178).

These works investigate human-preference alignment and scenario-adaptive conversational interaction. Our contribution is complementary but distinct: we evaluate whether an existing voice agent can infer and execute a role-implied behavioral policy without an explicit instruction. We will incorporate both works into Sec. 2 and clarify this distinction. We will also state that DSB-IFEval is a controlled diagnostic benchmark rather than a universal model of human conversational preferences.

Reviewer_yi4oOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DSB-IFEval to test whether FD voice agents infer appropriate floor-management behavior from persona descriptions rather than explicit instructions. It uses 240 scripted conversations across eight roles, expanded to 1,038 cases under default, explicit, persona-only, combined, and conflict conditions. A streaming orchestrator replays controlled speech events. Action-specific verifiers measure Instruction Adherence (IAS), while an LLM judge scores persona-consistent responses (PAS). It eval-ed 11 systems, and reports gaps between explicit and persona-only behavior and finds that models can adjust what they say without reliably adjusting when they speak. The appendices provide info on human validation and verifier-threshold analysis.

Critical strengths

The matched L1/L2/L3 design is nice. Keeping user audio fixed while changing explicit versus persona-implied instructions isolates a useful failure mode. Separating IAS from PAS also makes the content-versus-floor-control distinction concrete (Table 3).

The paper includes human checks of persona-to-action recoverability, action-level breakdowns, alternative judging, and verifier-threshold sensitivity. These are useful controls for a benchmark of this kind.

Critical weaknesses

I cannot reconcile some of the validation numbers with the stated protocol. Table 8 reports 85% accuracy but Cohen's kappa of 0.86 for ACCEPT OVERLAP and 0.87 for CONTINUE. For the same labels, kappa cannot exceed observed agreement. Table 3 also reports 35.5% L1 IAS for F-Actor, which cannot be obtained by averaging 240 binary outcomes as specified in Section 4 and Appendix G.4. If different denominators or aggregation rules were used, they need to be explained. Otherwise these are errors in the measurement evidence.

Questions for authors
  1. Can you provide the actual IAS denominators and Table 8 confusion matrices, and reconcile the reported accuracy, kappa, and pass rates?

  2. How would the main IAS and L1–L2 results change with sustained-overlap and order-sensitive readback checks?

  3. How were the expected instruction priorities independently validated for the 30 L4b safety-conflict cases?

1 comment
AuthorsOpenReview ↗

Official Response to Reviewer

We thank the reviewer for carefully examining the statistical reporting. We address the reviewer concerns below:

1. Human-validation statistics: Accuracy versus Cohen's κ (Table 8)

The reviewer's observation assumes that accuracy and Cohen's κ are computed against the same binary reference-label vector. Our human-validation study uses different reference constructions for these two statistics, a distinction that was not sufficiently explicit in Appendix A.2.

Specifically, each of the 360 evaluated responses was independently assessed by three blinded human annotators (1,080 judgments in total).

  • Accuracy/F1: Computed by comparing deterministic verifier decisions against each of the three individual annotators for each response, then macro-averaged for reporting in Table 8.
  • Verifier–human Cohen's κ: Computed against the majority-vote human label for each response.

Thus, accuracy measures agreement with individual human annotators, whereas verifier–human κ measures agreement with a consensus reference. The inequality cited in the review applies to κ and observed agreement computed from identical paired labels; it cannot be applied directly to these differently constructed statistics. We will explicitly document these aggregation formulas in Appendix A.2 and update the caption of Table 8 to distinguish mean individual-annotator accuracy from majority-reference Cohen's κ. Importantly, this clarification does not change the underlying human-validation protocol: all responses were evaluated by independent, blinded annotators using predefined behavioral criteria.

2. Clarification of IAS aggregation and reporting precision (Table 3)

The key distinction is between unique benchmark cases and independent evaluation runs. Appendix G.4 (Table 30) reports 240 unique cases for each of L1–L4a. Our evaluation protocol involves 10 independent runs per case, and the reported IAS is the mean of the 10 run-level scores. Consequently, each 240-case condition contributes 2,400 binary evaluation outcomes, yielding a score resolution of approximately 0.0417 percentage points for a single run. For example, the questioned F-Actor L1 IAS of 35.5% is mathematically attainable under this aggregation procedure (852/2,400 = 35.5%). Thus, the reported precision does not imply a non-binary verifier. Repeated evaluation also accounts for stochastic variation in model behavior under identical conversational stimuli. We will explicitly document the 10-run aggregation procedure and distinguish unique benchmark cases from execution counts in Sec. 4 and Appendix G.4 to eliminate this ambiguity.

3. Independent validation of instruction priorities in L4b safety conflicts

We thank the reviewer for raising this question. The 30 L4b cases are deliberately restricted to six safety-eligible scenario types across three roles: ER triage nurse, grief counselor, and 911 dispatcher (Appendix G.4, Table 30).

Unlike L4a, where an explicit conversational preference should prevail, L4b introduces a conflict with a role-critical safety obligation, such as intervening on an urgent clinical red flag despite an instruction not to interrupt. Importantly, the intended priorities were specified during benchmark construction, independently of model responses or evaluator judgments. The submitted paper additionally provides two complementary human-validation controls:

  • H1 — Persona-to-behavior validation (Appendix A.1, Table 7): Five independent, blinded annotators evaluated each of the 240 base conversations. Without seeing the target action or explicit instructions, they inferred the appropriate role-consistent behavior. Target agreement ranged from 69–87%, with Fleiss' κ of 0.69–0.91, supporting the independent recoverability of role-implied behavioral expectations.

  • H3 — Conflict-judgment validation (Appendix A.3, Table 11): Human annotators independently classified observed conflict-resolution outcomes. Agreement between human-majority labels and the two automated judges reached 96.7% and 97.5%, supporting the reliability of the conflict-outcome classification.

These controls validate the role-implied behavioral premise and the reliability of observed conflict classification, respectively. We will clarify these complementary validation stages in Appendix A and explicitly document the L4b priority criteria in Sec. 3.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.