- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper puts an additional evaluation aspect on implicit instruction following by defining role-grounded full-duplex behaviors. It compares different instruction and persona conditions, and measures the performance through timing- and persona-related metrics.
Critical strengths
Role-grounded full-duplex behaviors are still underexplored and worth being highlighted.
Critical weaknesses
Writing: I believe this work needs significant improvement in its writing to meet the ICLR bar.
-
Table 1 is never mentioned in the main text. The Introduction should further clarify the positioning of the proposed work in plain language.
-
Clearly mention that DSB-IFEval is the abbreviated form of the full name.
-
Lines 84 and 304 have missing citations.
-
Is “conversational probe” a widely used term in the field? This does not align with my initial understanding of “probing.” I want the authors to clarify whether the term was carefully selected by them rather than generated by AI.
-
There is no justification for shared/specific probes, and I could not find a clear difference between them. For example, the examples in lines 215–216 do not clearly support role-specific behaviors.
-
The descriptions of L1–L4 are too abstract to understand clearly. A concrete example would help readers.
-
Important information is included only in appendices that are insufficiently connected to the main text. For example, Table 30 shows that the experimental setups for L0 and L4b differ from those of the other conditions. Reviewers should not have to discover this information accidentally. The authors should reorganize the manuscript and address these details in the main text.
Motivation
- It is not very clear why the authors position DSB-IFEval as implicit instruction following, as it addresses explicit instructions with persona behaviors. This could be called “instruction following with a persona,” but not “implicit instruction following.”
Resource Novelty
- If the proposed benchmark is intended for future use, the implicit behaviors associated with each persona should have a generalizable basis. However, all behaviors are arbitrarily defined without any evidence. The current setup, which defines the behaviors first and then asks humans whether they agree, is not a grounded approach. Previous works, such as https://arxiv.org/abs/2506.21463 and https://arxiv.org/abs/2607.26178, have already attempted to align full-duplex behaviors with human preferences. The authors also do not provide evidence that the listed role behaviors are sufficiently diverse. For a reusable benchmark, sufficient justification is required.
Questions for authors
N/A
3 comments
Official Review Response Continued
5. Differences in experimental coverage across conditioning levels
These differences are intentional, but we agree they should be visible in the main paper. The benchmark contains 240 unique conversations (8 roles × 6 scenarios × 5 instances), expanded into 1,038 evaluation cases:
| Conditions | Cases per condition | Purpose |
|---|---|---|
| L0 | 48 | Default behavior, one instance per role–scenario cell |
| L1, L2, L3, L4a | 240 each | Full matched evaluation grid |
| L4b | 30 | Six safety-eligible cells × five instances |
Thus, L1–L3 support the central matched explicit-versus-implicit comparisons, whereas L0 is a smaller baseline and L4b a targeted safety-conflict evaluation. We will move these counts into Sec. 3.2, update Table 3's caption, and explicitly document the scoring denominators and paired comparison sets, with the complete selection rules remaining in Appendix G.4 (Table 30).
6. Why DSB-IFEval evaluates implicit instruction following
We respectfully disagree that our benchmark is simply "instruction following with a persona." The key distinction is whether the behavioral policy is explicitly specified or must be inferred.
- L1 (Explicit): The model receives a direct instruction specifying the required floor-management behavior.
- L2 (Implicit): The model receives only a persona description. It must infer the appropriate behavior from the role and conversational context, then execute it in real time.
- L3 (Combined): The implied rule is restated explicitly, allowing us to examine whether additional behavioral specification improves execution.
Crucially, the user-side audio and timing are held fixed across these conditions. This controlled comparison distinguishes the challenge of inferring a policy from executing an explicitly provided one. We also prevent personas from trivially restating the target instructions through imperative-leakage and lexical-overlap checks (Appendix B.6). Our results demonstrate why this distinction matters. For example, Gemini-3.8-Live's PAS increases from 32.8 to 66.8 under persona conditioning (L0→L2), whereas IAS remains approximately unchanged (29.0→28.6). Similarly, StepAudio-3-Realtime's PAS increases from 29.0 to 68.7, without improved IAS (35.5→33.5). These findings expose a meaningful gap between adopting persona-consistent language and translating a persona into appropriate real-time floor behavior. We will make this definition explicit in Secs. 1 and 3.1 and emphasize L2 as the primary implicit-instruction condition, with L1/L3/L4 serving as diagnostic comparisons.
7. Human grounding of persona-to-behavior mappings
We respectfully disagree that the role behaviors are arbitrarily defined without independent evidence. The submitted paper already includes a blinded human-validation study specifically addressing this concern (Appendix A.1, H1). Annotators received the L2 persona and user scenario but were not shown the explicit L1 instruction or the benchmark's expected action. Importantly, annotators were not asked to agree with an author-provided rule; they inferred the behavior independently without seeing the intended answer. This directly addresses the concern that the validation merely confirms predefined mappings. We further acknowledge that independently recoverable behavior is not necessarily the universally preferred behavior in every real-world interaction. We will explicitly distinguish these concepts and discuss less-unambiguous roles, including the 911 dispatcher and simultaneous interpreter. We will bring the human-study methodology and key findings into Sec. 3.2, rather than leaving this evidence solely in Appendix A.1.
Official Review Response Continued
8. Behavioral diversity and relationship to prior research
Our eight roles were selected for behavioral contrast rather than occupational diversity alone. Table 2 defines their policies along five dimensions: interruption, backchanneling, silence tolerance, readback, and overlap handling. For example, prolonged silence may be appropriate for a meditation instructor but not for a dispatcher resolving urgent ambiguity. Similarly, an interpreter may continue through brief overlap, whereas a triage nurse may need to yield immediately. We will strengthen Sec. 3.2 with an explicit role-by-behavior comparison demonstrating these contrasts and connecting it to the matched-audio diagnostic in Table 6, where byte-identical user speech is evaluated under three role-specific behavioral targets.
We also appreciate the suggested references:
- Aligning Spoken Dialogue Models from User Interactions (arXiv:2506.21463).
- DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues (arXiv:2607.26178).
These works investigate human-preference alignment and scenario-adaptive conversational interaction. Our contribution is complementary but distinct: we evaluate whether an existing voice agent can infer and execute a role-implied behavioral policy without an explicit instruction. We will incorporate both works into Sec. 2 and clarify this distinction. We will also state that DSB-IFEval is a controlled diagnostic benchmark rather than a universal model of human conversational preferences.
Official Review Response
We thank the reviewer for the detailed feedback and for recognizing that role-grounded full-duplex behavior is an underexplored research problem. We appreciate the valuable feedback and address each concern below.
1. Writing, presentation, and positioning of the benchmark
We thank the reviewer for providing valuable feedback to improve the presentation of the paper. We will make the following presentation improvements:
2. Use of the term "conversational probe" and distinction between shared and role-specific probes
We use the term probe in the diagnostic sense of a controlled conversational event designed to elicit a measurable behavioral decision. We do not intend to claim that "conversational probe" is an established term or a new probing methodology. As authors, we take responsibility for the choice and use of this terminology. To avoid confusion, we will replace it with controlled conversational test scenarios where appropriate in Sec. 3.2.
3. Justification for shared and role-specific scenarios
The two scenario types serve distinct experimental purposes:
For example, Appendix B.3 (Table 16) contains:
Thus, the scenarios test different behavioral requirements, not merely different dialogue topics. We will add this distinction and a representative table to Sec. 3.2, while retaining the complete specifications in Appendix B.3.
4. Concrete explanation of the L0–L4 protocols
We agree that an example will make the conditioning levels easier to understand. We will move the grief-counselor example from Appendix B.5 into Sec. 3.1 and revise Figure 2 accordingly.
Consider a grief counselor interacting with a user who pauses during a difficult disclosure:
L4b separately tests safety-critical conflicts in which a role-implied obligation should override an explicit directive. The comparison therefore evaluates whether a model can infer and execute the same policy when it is communicated implicitly rather than explicitly.