- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Relationship Permission Boundary (RPB), a paired counterfactual benchmark for full-duplex speech agents. Each pair keeps the audio before the decision point fixed and changes only the relationship between the interlocutors. The benchmark tests whether a model changes its conversational action when the relationship should matter, and keeps it unchanged when it should not. They evaluate three native audio systems, and find that all three models rarely adapt in the correct direction. Besides, the authors propose a hot-plug adapter on a frozen MiniCPM-o-4.5 backbone.
Critical strengths
-
The paper addresses an interesting and underexplored question: whether the relationship between interlocutors should change a speech agent’s conversational action at the same acoustic moment. To my knowledge, no existing full-duplex benchmark targets this capability.
-
The paired counterfactual design is sound. It holds the decision-preceding audio fixed and varies only the relationship description, so its effect is isolated cleanly. The metrics are well motivated. Directional Correct Adaptation (DCA) and the invariant controls separate correct, relationship-driven adaptation from local action accuracy and from generic prompt sensitivity. The constant-policy baselines also show clearly why condition-level accuracy alone is misleading.
Critical weaknesses
- The validity of the benchmark is not established. The Gold annotation process has two steps: two audio-capable model families propose candidate labels, and the authors then review and ratify the final Gold. However, the paper reports no inter-annotator agreement, and the reviewers could see the model proposals. The paper also does not name the model families used to propose labels, so it is unclear whether they overlap with the evaluated systems. Without independent annotation or a human ceiling, it is hard to tell whether the low DCA reflects model limitations or ambiguity in the reviewed adaptation directions themselves.
- The benchmark is small. The strong set contains only 79 pairs, so the reported DCA differences between models correspond to just a few pairs (0–3).
- Ecological validity is questionable. Pairing the same audio with different relationship descriptions may produce unnatural combinations. For example, the speakers’ tone or wording may already suggest a particular relationship that contradicts the assigned description.
- Key evaluation details are missing. The main text does not explain how the relationship information is provided to the native audio models (e.g., as a system prompt, text input, or speech). It also does not explain how each model’s native output is mapped to the four actions.
- The proposed hot-plug module provides limited evidence. The four relation states deterministically map to the four actions, and training and confirmation share the same mapping. The input therefore already encodes the answer, and a simple lookup table would achieve a perfect score. The relation-blind control is likewise guaranteed to fail, since it is denied exactly the information that determines the label. Moreover, the adapter seems not use the waveform, and acoustic-feature nulling leaves the results unchanged.
Questions for authors
Suggestions for the authors:
- Including a few concrete benchmark examples in the main text would help readers follow the setup.
- Several tables referenced in the main text appear only in the appendix (e.g., Section 6.3 relies on Table 6). Moving these tables into the main text would improve readability.
- Some notation in Section 3.2 is used without being defined, for example the episode index e and the relationship condition r. Section 4 would also benefit from more textual explanation or worked examples of the metric definitions, especially DCA (Eq. 4).
2 comments
Follow-up on our rebuttal
Dear Reviewer, Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our point-by-point responses, clarifications, and additional results address your main concerns. As the discussion period is nearing its end, please let us know if any key concerns remain. We would be happy to provide further clarification. If our responses have addressed your main concerns, we would be grateful if you could reconsider your assessment and update your rating as appropriate. Thank you for your consideration.
Response to Reviewer 4vLa: Reference Validity, Ecological Scope, Interface Details, and Adapter Interpretation
Thank you for emphasizing reference validity, ecological scope, and the need to distinguish primary evidence from supplementary implementation studies. These comments helped us sharpen both the empirical scope and the paper's presentation.
Response to W1-W2: Reference validity and scale
The evidence scope changes as follows:
We report condition-level agreement, pair-split agreement, exact-agreement robustness, DCA reachability, and raw exports. The conditional leave-one-rater-out analysis is presented as a robustness check, not a population human ceiling.
The revision also identifies the legacy proposal source that was previously unnamed: the archived reference used Qwen/Gemini family probability judgments, with conflict adjudication on a subset. These are not the same systems as the evaluated PersonaPlex, MiniCPM-o, and Freeze-Omni baselines. More importantly, the revised primary Gold is rebuilt from the three independent human exports and is never pooled with the legacy proposals; the latter are retained only to explain historical panel selection.
Response to W3: Ecological validity
Your comment usefully distinguishes a controlled counterfactual from naturally occurring relationship change. Fixing audio while varying a compact relationship description isolates relationship conditioning, but some assigned relationships may be less compatible with the recorded wording or tone. Annotators judge the supplied relationship together with the pre-cutoff context; this is not a separate blinded audio-relationship compatibility rating. We therefore scope conclusions to the controlled counterfactual setting and avoid claims about naturally occurring relationship shifts or universal social norms. A separate compatibility audit remains useful future validation.
Response to W4: Interface and mapping details
Each native system receives the same decision-preceding content and a shared structured relationship description containing role, familiarity, authority, and task responsibility; the prompt never names a target action. The revision distinguishes response-derived heuristic labels from Freeze-Omni's speaking-state proxy and supplies the prompts, streaming code, observation windows, and extractor implementations.
Response to W5: Hot-plug module
The deterministic mapping, lookup-equivalent controls, and acoustic-null result show that this experiment is best interpreted as a contract-execution check rather than a learned remedy. The adapter is therefore removed from the paper's contribution list and reported only in the supplementary case study.
Presentation revisions
We add a concrete representative pair, define e and r at first use, expand the explanation after the DCA equation, and retain the failure decomposition and primary factorial tables in the main text. Extractor-version sensitivity and detailed diagnostics move to the appendix, keeping the scientific main text within nine pages while preserving the latest figures and tables.