acceptodds
Under review as a conference paper at ICLR 2027

Being Reasonable Is Not Enough: Measuring Selective Relationship Conditioning in Full-Duplex Speech Agents

Abstract

Full-duplex speech agents must decide not only whether a conversational moment permits a response, but whether the same moment warrants a different action for a friend, supervisor, or task owner. Ordinary action accuracy cannot isolate this capability because a locally acceptable default may ignore the relationship condition. We introduce the Relationship Permission Boundary (RPB), a paired counterfactual benchmark that holds decision-preceding audio fixed while changing only the interlocutor relationship. Directional Correct Adaptation (DCA) requires both sides to be acceptable and any action change to follow the reviewed direction. On 79 human-reviewed strong pairs, three native audio systems obtain only 0–3.8% DCA although condition-level acceptable accuracy reaches 31.6–75.3%; a constant WAIT policy reaches 88.0% Acceptable with DCA 0. A concise training analysis identifies three recurring shortcuts: local fit without selective transfer, direction without invariant safety, and safety through zero-change collapse. We then train a detachable relationship FiLM router, speech-activity feature encoder, and paired action head using frozen MiniCPM-o-4.5 text scores. On a participant-isolated AMI Train-only endpoint, all three seeds obtain DCA 1, balanced safe utility 1, coverage 0.444444, exact 1, and zero harm or invalid change. The frozen base has DCA 0 and utility 0.638889; an equal-capacity relation-blind control has DCA 0 and utility 0.666667. Both nine-meeting recording-blocked tests give p = 0.0020999. Training and confirmation share a deterministic four-state/action mapping, so this is a policy-contract result, not evidence of learned human permission boundaries or audio-conditioned adaptation. A post-hoc exploratory audit supports dependence on supplied state and side binding; acoustic-feature nulling is unchanged (p = 1.0). The contribution is an auditable benchmark specification, a shortcut diagnosis, and bounded evidence of structured-state use.

Primary area:
datasets and benchmarks
Keywords:
full-duplex speech agents, relationship conditioning, counterfactual evaluation, social interaction, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4vLaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Relationship Permission Boundary (RPB), a paired counterfactual benchmark for full-duplex speech agents. Each pair keeps the audio before the decision point fixed and changes only the relationship between the interlocutors. The benchmark tests whether a model changes its conversational action when the relationship should matter, and keeps it unchanged when it should not. They evaluate three native audio systems, and find that all three models rarely adapt in the correct direction. Besides, the authors propose a hot-plug adapter on a frozen MiniCPM-o-4.5 backbone.

Critical strengths
  1. The paper addresses an interesting and underexplored question: whether the relationship between interlocutors should change a speech agent’s conversational action at the same acoustic moment. To my knowledge, no existing full-duplex benchmark targets this capability.

  2. The paired counterfactual design is sound. It holds the decision-preceding audio fixed and varies only the relationship description, so its effect is isolated cleanly. The metrics are well motivated. Directional Correct Adaptation (DCA) and the invariant controls separate correct, relationship-driven adaptation from local action accuracy and from generic prompt sensitivity. The constant-policy baselines also show clearly why condition-level accuracy alone is misleading.

Critical weaknesses
  1. The validity of the benchmark is not established. The Gold annotation process has two steps: two audio-capable model families propose candidate labels, and the authors then review and ratify the final Gold. However, the paper reports no inter-annotator agreement, and the reviewers could see the model proposals. The paper also does not name the model families used to propose labels, so it is unclear whether they overlap with the evaluated systems. Without independent annotation or a human ceiling, it is hard to tell whether the low DCA reflects model limitations or ambiguity in the reviewed adaptation directions themselves.
  2. The benchmark is small. The strong set contains only 79 pairs, so the reported DCA differences between models correspond to just a few pairs (0–3).
  3. Ecological validity is questionable. Pairing the same audio with different relationship descriptions may produce unnatural combinations. For example, the speakers’ tone or wording may already suggest a particular relationship that contradicts the assigned description.
  4. Key evaluation details are missing. The main text does not explain how the relationship information is provided to the native audio models (e.g., as a system prompt, text input, or speech). It also does not explain how each model’s native output is mapped to the four actions.
  5. The proposed hot-plug module provides limited evidence. The four relation states deterministically map to the four actions, and training and confirmation share the same mapping. The input therefore already encodes the answer, and a simple lookup table would achieve a perfect score. The relation-blind control is likewise guaranteed to fail, since it is denied exactly the information that determines the label. Moreover, the adapter seems not use the waveform, and acoustic-feature nulling leaves the results unchanged.
Questions for authors

Suggestions for the authors:

  1. Including a few concrete benchmark examples in the main text would help readers follow the setup.
  2. Several tables referenced in the main text appear only in the appendix (e.g., Section 6.3 relies on Table 6). Moving these tables into the main text would improve readability.
  3. Some notation in Section 3.2 is used without being defined, for example the episode index e and the relationship condition r. Section 4 would also benefit from more textual explanation or worked examples of the metric definitions, especially DCA (Eq. 4).
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4vLa: Reference Validity, Ecological Scope, Interface Details, and Adapter Interpretation

Thank you for emphasizing reference validity, ecological scope, and the need to distinguish primary evidence from supplementary implementation studies. These comments helped us sharpen both the empirical scope and the paper's presentation.

Response to W1-W2: Reference validity and scale

The evidence scope changes as follows:

Evidence itemReviewed versionRevision
Headline strong subset79 pairs181-pair primary directional split
Independent model-blind labelsNo benchmark-wide result3 raters × 1,022 conditions
Prospective coverageLimited endpointPersonaPlex over all 511 pairs

We report condition-level agreement, pair-split agreement, exact-agreement robustness, DCA reachability, and raw exports. The conditional leave-one-rater-out analysis is presented as a robustness check, not a population human ceiling.

The revision also identifies the legacy proposal source that was previously unnamed: the archived reference used Qwen/Gemini family probability judgments, with conflict adjudication on a subset. These are not the same systems as the evaluated PersonaPlex, MiniCPM-o, and Freeze-Omni baselines. More importantly, the revised primary Gold is rebuilt from the three independent human exports and is never pooled with the legacy proposals; the latter are retained only to explain historical panel selection.

Response to W3: Ecological validity

Your comment usefully distinguishes a controlled counterfactual from naturally occurring relationship change. Fixing audio while varying a compact relationship description isolates relationship conditioning, but some assigned relationships may be less compatible with the recorded wording or tone. Annotators judge the supplied relationship together with the pre-cutoff context; this is not a separate blinded audio-relationship compatibility rating. We therefore scope conclusions to the controlled counterfactual setting and avoid claims about naturally occurring relationship shifts or universal social norms. A separate compatibility audit remains useful future validation.

Response to W4: Interface and mapping details

Each native system receives the same decision-preceding content and a shared structured relationship description containing role, familiarity, authority, and task responsibility; the prompt never names a target action. The revision distinguishes response-derived heuristic labels from Freeze-Omni's speaking-state proxy and supplies the prompts, streaming code, observation windows, and extractor implementations.

Response to W5: Hot-plug module

The deterministic mapping, lookup-equivalent controls, and acoustic-null result show that this experiment is best interpreted as a contract-execution check rather than a learned remedy. The adapter is therefore removed from the paper's contribution list and reported only in the supplementary case study.

Presentation revisions

We add a concrete representative pair, define e and r at first use, expand the explanation after the DCA equation, and retain the failure decomposition and primary factorial tables in the main text. Extractor-version sensitivity and detailed diagnostics move to the appendix, keeping the scientific main text within nine pages while preserving the latest figures and tables.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer, Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns. As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification. If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate. Thank you for your consideration.

Reviewer_FFzVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Relationship Permission Boundary (RPB), a paired benchmark for whether FD speech agents adjust turn-taking actions when only the interlocutor relationship changes. Each pair holds the decision-preceding audio fixed. Set-valued acceptable actions and Directional Correct Adaptation (DCA) distinguish locally valid responses from changes in the expected direction, while invariant pairs penalize unnecessary changes. On 79 strong pairs, three native speech systems achieve 0–3.8% DCA, despite higher action acceptability. A constant WAIT policy scores 88% acceptability with no adaptation. A separate FiLM-style adapter reaches perfect scores on a fixed structured-state policy contract, which the paper distinguishes from learning real human permission boundaries.

Critical strengths

The paired test is a useful way to distinguish actual relationship sensitivity from an action prior. Constant WAIT outperforms the tested models on local acceptability but has zero DCA. Aggregate accuracy would hide both failures.

Set-valued acceptable actions and invariant controls are sensible. The paper also reports the negative cross-corpus transfer result rather than presenting its positive policy-contract experiment as a solution to real-audio adaptation.

Critical weaknesses

The benchmark's most important labels have limited independent validation. Gold actions are proposed by models and ratified by authors, without benchmark-wide blinded annotator agreement or a human align. The native-speech-to-action mapping also lacks perceptual validation. On only 79 strong pairs, near-zero DCA could partly reflect debatable permission boundaries or extraction errors. In particular, an unchanged action may still be acceptable in both conditions.

The adapter result does not show that the model learned relationship-dependent speech behavior. Its four input policy states deterministically identify the target actions, and the same mapping is used for training and confirmation. Acoustic-feature nulling changes nothing, and the separate preregistered cross-corpus experiment does not improve over the frozen base. Perfect contract scores therefore demonstrate a fixed state-to-action lookup, not generalization to unseen permissions or audio contexts.

Questions for authors
  1. Are there model-blind human judgments for the strong pairs and independently audit the native-response action extractor? Does low DCA persist on high-agreement cases?

  2. In how many strong pairs is changing action necessary for acceptability, rather than only for the preferred direction? Can the core inputs and scorer be released for verification?

2 comments
AuthorsOpenReview ↗

Response to Reviewer FFzV: Model-Blind Human Validation, Necessary Action Changes, and Adapter Scope

Thank you for proposing checks that distinguish reference validity from native-action measurement. These two audits answer different questions, and the revision now reports them separately.

Response to Q1: Model-blind judgments, high-agreement cases, and extractor audit

The permission reference is now independently validated: all 1,022 conditions have three independent model-blind human judgments, with raw exports and agreement statistics released. On the exact-agreement strong subset covered by all three systems:

SystemExact-agreement strong DCA
PersonaPlex14/27
MiniCPM-o1/27
Freeze-Omni1/27

The independent reference supports a model-specific, reference-scoped conclusion rather than one uniform statement across systems.

The 14/48 explicit-action execution control is an informative diagnostic, but it is not a blinded listening confusion matrix. We accordingly scope system claims to extractor-derived actions for complete model-interface-extractor configurations and release all prospective response WAVs to enable the requested listener audit.

Response to Q2: When is changing action necessary?

Strong-pair categoryCountMeaning of change
No action acceptable on both sides25/181Necessary for both predictions to remain acceptable
Shared acceptable action, no shared preferred action109/181Preference-directional, not required for local acceptability
At least one shared preferred action47/181Unchanged behavior can remain locally defensible

We add these counts to the main text and clarify that DCA measures adherence to the aggregate direction, not the social wrongness of every unchanged output.

Response to the adapter concern

Your concern usefully distinguishes contract execution from learned relationship-dependent speech behavior. The same state/action mapping is used in training and confirmation, acoustic nulling changes nothing, and lookup, relation-only linear, and backbone-free tiny-MLP controls match the adapter. We therefore remove the adapter from the contribution list and retain it only in the supplementary case study as a contract-execution sanity check and negative-transfer boundary.

The supplementary ZIP includes condition inputs and executable input metadata, raw annotation exports, the aggregate and scorer, frozen action records, runner snapshots, and deterministic verification tools, subject to the licensed-audio and runnable-environment boundaries described in the paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer, Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns. As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification. If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate. Thank you for your consideration.

Reviewer_ZHXZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Relationship Permission Boundary (RPB) benchmark to evaluate whether speech agents selectively adapt conversational actions to the interlocutor relationship. Each counterfactual pair preserves the decision preceding audio while changing the relationship description. The benchmark uses acceptable and preferred action sets over WAIT, BACKCHANNEL, TAKE_TURN, and INTERRUPT. Its Directional Correct Adaptation (DCA) metric requires both predictions to be acceptable and their change to follow a supported permission boundary or preference reversal. Invariant pairs test whether models change when they should remain stable.

The core contains 511 pairs, with the main native model evaluation using 79 strong pairs and 188 invariant controls. Three systems achieve only 0-3.8% DCA on the strong set, while an always-WAIT policy achieves 88% condition level acceptability. The paper also presents a detachable policy module that achieves perfect performance on a separate, 36 pair AMI derived endpoint. As the authors explicitly explain, this endpoint shares a deterministic 4 state-to-action mapping with training, and performance is unchanged when acoustic features are removed. The contributions are therefore a paired evaluation framework, an empirical diagnosis under its reviewed labels, and a narrowly scoped demonstration of learning a supplied policy contract.

Critical strengths

1. The evaluation design separates locally acceptable behavior from selective relationship use.

Holding the decision preceding audio fixed while changing the relationship description is a useful way to test whether a model responds to that information. Combining set valued acceptability, directional adaptation, and invariant controls avoids reducing conversational competence to either a single mandatory action or unrestricted prompt sensitivity.

The constant policy results make the motivation concrete: always WAIT obtains 88% acceptable accuracy while necessarily achieving zero DCA (Sec. 6.2 and Table 5). The distinction matters because a system can perform well under an aggregate action metric without demonstrating the conditional behavior the application requires. This is the paper's strongest contribution and could support a valuable benchmark if its measurement assumptions are validated.

2. The diagnostic analysis and transparent reporting make the results informative within their stated scope.

The decomposition in Table 6 distinguishes valid changes, invalid changes, and unchanged behavior, giving more information than a single adaptation score. The invariant control evaluation further shows why increased sensitivity to relationship wording is insufficient. The additional PersonaPlex seeds support the observation that its low DCA persists across the tested runs, despite changing action marginals.

I also appreciate the explicit reporting of the unsuccessful cross corpus transfer result, the model assisted annotation provenance, and the limited interpretation of the perfect policy contract result. These disclosures allow the reader to distinguish what the experiments establish from what remains untested. They strengthen the paper's credibility and make a focused revision more feasible.

Critical weaknesses

1. The relationship permission labels need independent validation before the central behavioral diagnosis is sufficiently convincing.

The benchmark depends on judgments about which actions are acceptable or preferred under a relationship condition, and whether changing that condition warrants a directional change. Section 3.2 states that model families propose labels and dataset authors review and ratify them. The 96 item, 4 person review subset does not supply independent validation. It is training only, reviewers saw model answers, and raw reviewer level exports are unavailable. Sec. 3.2 and 9 explicitly acknowledge the absence of independent benchmark-wide agreement measurements and a human ceiling.

Author review is meaningful, but it does not establish how consistently independent listeners would endorse the resulting permission profiles. This matters particularly for the 79 strong pairs that drive the headline DCA result. The manuscript gives the category counts without a sufficiently operational rule for distinguishing strong pairs from the 244 boundary pairs.

There is also an important interpretation issue. DCA requires a change, so unchanged predictions can receive zero even when both actions are acceptable. This is consistent with the metric's intended purpose. However, it makes independent evidence that a change is warranted especially important. The always WAIT result demonstrates that acceptability alone cannot establish conditioning. It does not, by itself, establish that every unchanged decision is socially inappropriate.

Suggested improvement: Provide the annotation rubric and explicit strong/boundary/invariant assignment rules, with representative complete examples. A bounded, independently conducted audit of existing strong and invariant pairs, with annotators initially unaware of model proposals, would materially strengthen the paper. Report agreement separately for acceptable actions, preferred actions, and the judgment that a directional change is warranted. Human paired performance under the same information constraints, together with sensitivity to reasonable adjudication choices, would help establish how the low model scores should be interpreted. Such an audit can target the existing benchmark without requiring a substantially expanded dataset.

2. The native action measurement is insufficiently validated and cannot currently be independently reproduced from the described submission bundle.

Sec. 5 states that native system responses are mapped into the four actions, but the manuscript does not provide enough operational detail to evaluate that transformation. For example, the observation window determining WAIT and the timing/content criteria separating BACKCHANNEL, TAKE_TURN, and INTERRUPT are central to the measurement. Sec. 8 explicitly identifies perceptual validation of the action extractor as unfinished.

All headline native-model results depend on this mapping. Consequently, the current evidence does not adequately separate relationship-conditioning failures from possible effects of response latency, output style, interface handling, or action classification. Freeze Omni's strong tendency toward INTERRUPT may reflect its actual behavior, but the interpretation would be considerably more persuasive with independently checked examples and model specific extraction accuracy.

This concern is compounded by the Reproducibility Statement, which says that the submission bundle excludes the core dataset, the three native adapter sources, and the runtime frozen scorer source. The authors appropriately disclose that exact model evaluation reruns are therefore unavailable from the bundle. For a benchmark contribution, these omissions substantially limit both independent verification and reuse. Integrity manifests document artifact identity, but do not provide the missing evaluation instances or executable measurement procedure.

Suggested improvement: Specify the native inference and action extraction protocols, including prompts, observation windows, timing rules, and ambiguous response handling. Audit a stratified subset of existing outputs across all three models and all four actions using listeners unaware of model identity. Report the extraction confusion matrix or agreement and recompute the principal paired results using the checked labels. Provide the core evaluation records, or lawful reconstruction instructions containing source identifiers and cutoffs, together with relationship conditions, Gold labels, inference wrappers, and the fixed scorer. These steps would directly strengthen the existing empirical claim.

Questions for authors
  • What precise rules distinguish strong, boundary, and invariant pairs? Can a strong pair have a shared preferred action across both conditions, and, if so, how is the requirement for directional change justified? Please provide complete examples and clarify what independent evidence, if any, supports these assignments.

  • How are native outputs converted into WAIT, BACKCHANNEL, TAKE_TURN, and INTERRUPT? In particular, what observation horizon distinguishes waiting from a delayed response, and how are backchannels distinguished from taking or interrupting the floor? Would the principal DCA and invariant-control findings remain after a blinded manual audit of the existing outputs?

  • Can the core evaluation instances or deterministic reconstruction instructions, native interface wrappers, and frozen scorer be made available for inspection and rerunning? Which specific restrictions prevent their inclusion, and what reproducible access route is available?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZHXZ: Independent Reference Validation, Native Action Measurement, and Reproducibility

Thank you for highlighting the importance of independent reference validation and perceptual action measurement. Your comments helped us separate three questions more clearly: whether the permission reference is reliable, whether systems change selectively with relationship context, and whether the extracted actions have been perceptually validated.

W1: Independent validation and operational split rules

We added a complete three-rater, model-blind audit. Three non-author native-English annotators independently labeled all 1,022 conditions without seeing peer judgments, the prior reference, or evaluated-system answers. We release the raw exports, protocol, aggregate, agreement report, and exact aggregation code. Chosen-action nominal alpha is 0.767; mean pairwise Jaccard is 0.919 for Acceptable sets and 0.880 for Preferred sets.

A pair is strong if its highest-probability action changes or, with the primary action unchanged, it exhibits a bidirectional rank reversal of at least 0.10. A pair is invariant if its Acceptable and Preferred sets are unchanged and total variation is below 0.10; remaining non-strong variations are boundary pairs. This yields 181 strong, 189 boundary, and 141 invariant pairs. Enumerating all 16 ordered output pairs shows that DCA is attainable on 180/181 strong pairs; we retain the single unattainable pair.

Strong-pair categoryCountInterpretation
No shared acceptable action25/181Change is required for both sides to remain acceptable
Shared acceptable, no shared preferred action109/181Change is preference-directional
At least one shared preferred action47/181Unchanged behavior can remain locally defensible

Thus, DCA=0 does not imply that an unchanged output is necessarily socially impermissible.

The revision adds a complete worked pair. The same audio is paired with close friends with peer authority versus a low-familiarity formal relationship with supervisory asymmetry. The aggregate gives A0=P0={TAKE_TURN} and q0(TAKE_TURN)=0.745, versus A1={WAIT,BACKCHANNEL,TAKE_TURN}, P1={WAIT}, and q1(WAIT)=0.507. TAKE_TURN→WAIT is acceptable on both sides, hits both Preferred sets, and has WAIT∉A0; hence CS=1 and DCA=1.

W2: Native output-to-action measurement

We now define all four actions operationally and release model-specific prompts, windows, and mapping code. WAIT is no agent speech in the scoring window; BACKCHANNEL is a short feedback token that does not claim the floor; TAKE_TURN begins a substantive turn after floor release; and INTERRUPT begins while the interlocutor still holds the floor. For the prospective PersonaPlex study, the horizon is the final 3,000 ms before the cutoff. A named short feedback token, or a response of at most three words and 1,200 ms, is BACKCHANNEL; substantive responses are separated into INTERRUPT versus TAKE_TURN by floor release. Freeze-Omni retains a speaking-state proxy and is labeled accordingly.

We added a 48-trial explicit-action execution control:

Requested actionMatchTotal
WAIT512
BACKCHANNEL312
TAKE_TURN212
INTERRUPT412
Total1448

This diagnostic does not replace a blinded listener confusion matrix. We therefore scope system numbers to extractor-derived results for complete model-interface-extractor configurations, rather than claiming listener-verified audible behavior. All prospective WAVs are retained for the suggested stratified audit.

W3: Reproducibility and release

The supplementary ZIP contains the complete 511-pair/1,022-condition inventory, annotation protocol, three raw human exports, aggregate, frozen action records, scorer, verification scripts, executable input metadata, prompts and runner snapshots, factorial reports, and hash manifests. After extraction, python3 -B -m tools.relationship_duplex.verify_review_release verifies aggregation, joins, frozen-action rescoring, and directionality enumeration.

Frozen historical outputs and recovered runners support inspection and deterministic rescoring. Exact historical reruns remain bounded by licensed decision audio, unavailable historical response media, model weights/native environments, and the historical runtime-scorer source. The prospective PersonaPlex study provides stronger provenance through complete run records and 3,066 hash-verified response WAVs. The revision distinguishes replayable scoring and frozen records from unavailable historical end-to-end inference.

Reviewer_ZHXZOpenReview ↗

Official comment to authors

I thank the authors for their experiments and detailed responses. Unfortunately, I still think informativeness and soundness stand at 2, and so I will keep my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.