acceptodds
Under review as a conference paper at ICLR 2027

Revision-Bound Grounding for Stateful Tool Use in Open-Source Full-Duplex Voice Agents

Abstract

Full-duplex voice agents can listen, speak, reason, and act concurrently, but this creates a state-consistency problem: user intent may change while earlier reasoning or tool execution remains in flight, allowing stale evidence or actions to affect the interaction. We introduce *revision-bound grounding*, a causal interface that binds asynchronous evidence, action proposals, authorizations, and tool outcomes to their originating dialogue revision and action trace, admitting them only while they remain authoritative. We instantiate this interface in MoshiAgent, which decouples continuous speech from asynchronous reasoning and tool execution while maintaining explicit dialogue, action, and observed world state. We also construct a tool-grounded duplex corpus of conversations totaling hours and train MoshiAgent-Speech to incorporate delayed evidence during ongoing interaction. Across six audio-agent benchmark families and ToolRaceBench, a diagnostic benchmark for asynchronous state consistency, revision-bound execution with fixed MoshiAgent-Speech improves Full-Duplex-Bench v3 argument accuracy from to and strict from to , while increasing ToolRaceBench from to . The complete system achieves Macro on -Voice, surpassing GPT-Realtime-2.1 mini. These results show that reliable full-duplex tool use requires explicit causal alignment between evolving user intent, external actions, and observed outcomes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
full-duplex voice agents, revision-bound grounding, asynchronous reasoning, tool execution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XNJJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses asynchronous reasoning/tool-calling problem in full-duplex interaction scenarios. They build MoshiAgent and ToolRaceBench for the target task.

Critical strengths

This manuscript addresses realistic and speech-oriented constraints where users may easily change their intents while background tool calling starts with outdated information.

Critical weaknesses

I consider the writing incomplete in its current form for the following reasons:

  • Figure 1 is not referenced in the text.

  • Figure 2 is not fully explained in the main text. The manuscript discusses only the streaming speaker, leaving the other components unexplained. All terms and components shown in the figure should be described in the main text, with clear connections to the figure.

  • Confusing terms are being used, such as "semantic" in sec 3.2, and they did not mention about its deterministic rules and actual models. Basic formulations, like dialogue states, are not explained how it looks like.

  • Although ToolRaceBench is an important contribution of the manuscript, all details are sent to the appendix. Essential information should be included in the main text to make the manuscript self-contained.

Overall, I suggest significantly revising the writing by removing redundant components and clearly highlighting contributions.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer XNJJ

We thank the reviewer for the specific comments on the presentation.

W1. Fig. 1 is not referenced

Thank you for catching this. Sec. 1 described the alarm example of Fig. 1 but did not cite the figure. It now cites Fig. 1(a) for the failure case and Fig. 1(b) for our handling.

W2. Fig. 2 is not fully explained

We agree. Sec. 3.1 previously described only the Streaming Speaker and the grounding interface as a whole. A new paragraph in Sec. 3.1 ("System Components") follows the layout of the figure and explains every labeled component and arrow. It covers speech (Streaming Speaker, Request, Respond), reasoning (Reasoning Coordinator, Fast Reasoner, Deliberative Explorer, Launch, Exploration Goal, State), and execution (Action, the "Side-effecting action?" branch, Tool Executor, Write Control, User Confirmation, Results, Findings, Evidence Admission, Authoritative Session State).

W3. The term "semantic", the deterministic rules, and the models

Sec. 3.2 stated that deterministic rules handle unambiguous acceptance and rejection and that a semantic classifier handles modifications and mixed intent, but it did not explain "semantic", give an example of the rules, or name the classifier. It now adds all three. Supersession is semantic because it depends on what the new speech means for the pending work, not only on the revision counter. For example, a backchannel ("mm-hmm") advances the revision but leaves earlier work valid, whereas "actually, make it 10" supersedes it. The rules decide clear consent or refusal (e.g., a bare "yes" or "no"). The classifier is the reasoning backend itself (Gemma-4-31B-it by default). It is invoked only for speech with a modification cue or mixed affirmative and negative words, and all other speech (e.g., a question) is unclear. Appendix A.2 gives the decision table and pseudocode.

W4. The dialogue state is not explained

We agree that a concrete description was missing. Sec. 3.2 introduced the dialogue state only as the user-side transcript with a revision counter. Sec. 3.1 now describes all three parts of St=(Dt,At,Wt)S_t = (D_t, A_t, W_t). DtD_t is the user transcript with the revision counter rr, AtA_t records each proposal, authorization, and tool invocation with its status and outcome, and WtW_t holds the observations and verified effects returned by tools. For example, in Fig. 3(a), "Actually, make it 10 a.m." advances rr and marks set_alarm(9) as superseded in AtA_t, while WtW_t still records 8 a.m. Appendix A.1 gives the full schema.

W5. ToolRaceBench details are only in the appendix

Sec. 5.1 described the construction, the scenario format, and what is evaluated, but not the size, categories, pass criterion, or scoring. It now reports the size (298 scenarios), the number of primary categories (nine, distributed as in Table 20 of Appendix E), the pass criterion, and the scoring.

W6. Redundancy and contributions

We removed a repeated set of result numbers from the introduction and several sentences in Sec. 5 that only restated table values. The first contribution now names what is new, namely the authoritative session state, proposal-relative semantic supersession, and the properties P1 to P4 with the runtime procedure that enforces them.

We hope the revision addresses the reviewer's concerns about the presentation. Since the missing details may also have made the method hard to assess, we note that Appendix A now gives the runtime specification. We would be glad to address any remaining concern about soundness and would appreciate it if the reviewer could take another look at the revised version.

Reviewer_NfP2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors train and a realtime agentic speech model for incorporating checks and balances for ongoing interaction, evaluated extensively on seven benchmark families. Seperately authors also release a training dataset and evaluation benchmark for evaluating tool usage.

Critical strengths

The problem is important and timely, in conversation with agents mistakes or misunderstandings can happen, and having a framework to fix these conversations before performing action is useful. I like the extensive evaluations over a variety of different types of (combinations of) models, including the evaluation on various diferent benchmarks. Based on this MoshiAgent scores state-of-the-art on many different use cases. Also good ablations, and clear and easy to follow writing.

Critical weaknesses
  1. I don't fully agree with the premise here, when adding an extra propose step in the pipeline before coming to an actual action, is that the authors would delay the user intent, resulting in a slower pipeline in the end (Figure 1). The user has to say another agreement, and there is more steps involved overall. More often than not it could be to have an user saying an intended ask on the first try. I believe this is a drawback of the method (slower time to finish task).

  2. I’m missing some ablations/evaluations with respect to the dataset and method. To benchmark also how this framework is applied to other realtime models. If the authors have this dataset of 599K conversations, and the authors train another open source model on it, would it also perform well or as well in terms of increase of performance when applied to MoshiAgent-Speech. So not just the reasoning backend, but a whole different model architecture.

  3. In my opinion the authors should have redacted the name Moshi in the paper. Calling the proposed interface MoshiAgent (and model MoshiAgent-Speech) kind of defeats the purpose of the double-blind review, knowing that the paper is probably affiliated with the well-known speech model Moshi and/or the work done by Kyutai.

Questions for authors
  1. I imagine for example when paying for a service or goods, at a certain amount of money it would be good to have this extra check for user intent in. And if the service or goods are free it might be useful to skip this extra evidence check. To summarize: for what kind of interaction would it be useful to have this extra evidence check (Figure 1) and when might it not be important to ask for extra confirmation?

  2. As the authors mentioned in related work that some speech works nowadays focus on decoupling the slower reasoning with the faster realtime aspect, I think its important to have some sort of comparison also in terms of latency. How would the strategy of the authors be in getting higher accuracy in tasks, but at what costs in latency the authors have, i.e. from start of the task how long it takes to have the task registered (not just turns)?

  3. What kind of types of tool calling are there in the data, like what in general can be seen as a tool call, is it the same as is the case in current foundation models? For example, can it run a python script within a tool as well?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NfP2

We thank the reviewer for the positive assessment and thoughtful questions.

W1 & Q2. The extra confirmation step and its latency cost

Fig. 1 is a simplified example, and its panel (b) contains a proposal and a "Yes" that panel (a) does not, which may suggest that our method adds an extra confirmation round. We clarify with three points.

First, this confirmation is not something only our method adds. The evaluation tasks themselves require it. The τ-Voice Airline and Retail policies require confirmation before any database update. The Airline policy reads, "Before taking any actions that update the booking database ..., you must list the action details and obtain explicit user confirmation (yes) to proceed." P3 binds this "yes" to the exact action, so a user whose first request is correct goes through the same turns. An extra exchange occurs when the user revises the request, which is when executing the earlier request would be incorrect.

Second, only side-effecting writes are gated. Read-only calls such as lookups go directly to the Tool Executor (the "Side-effecting action?" branch in Fig. 2).

Third, we measured the cost at the task level rather than in turns. On the tasks completed by both, the complete system takes on average 3.0 s longer on Retail and 4.7 s less on Telecom than the ablation without Write Control (Table 4, Appendix D). Since the confirmation gate takes effect in both domains, it adds almost no time, while it improves Pass@1 from 27.2 to 31.3 on Retail and from 13.2 to 15.8 on Telecom.

For turn-level latency, Table 1 compares MoshiAgent (184 ms smooth-turn, 388 ms interruption) with the decoupled-reasoning systems discussed in the related work, MoshiRAG (180 ms, 1020 ms) and DuplexOmni (246 ms, 510 ms).

Q1. When is the extra check useful, and when is it not?

We agree that this depends on the stakes, as in the reviewer's payment example. Confirmation is warranted for irreversible or costly actions (payments, bookings, cancellations) and less important for low-risk, reversible ones. Read-only calls are never confirmed. Which writes are gated is a deployment choice, which we now discuss in Sec. 3.3.

W2. Ablations on the dataset and method, and other real-time models

Dataset and method. Table 3 crosses the speaker (MoshiRAG, or MoshiAgent-Speech trained on our corpus) with the execution policy. With the runtime fixed, training on our corpus improves ToolRaceBench from 29.2 to 83.2 and τ-Voice from 18.1 to 32.1. With the speaker fixed, the runtime improves them from 34.2 to 83.2 and from 18.6 to 32.1.

Another real-time model. Applied to the original MoshiRAG without retraining, the runtime improves ToolRaceBench from 9.7 to 29.2 and τ-Voice from 16.9 to 18.1 (Table 3).

A different architecture trained on our corpus. We agree that this would be valuable, but we do not yet have such results, and we note this in the Limitations (Appendix F). The corpus is not specific to Moshi. It consists of role-separated stereo 24-kHz audio, word-level alignments, and factual references, and a target model needs a channel for injecting text evidence asynchronously.

W3. The name "Moshi"

The name refers to the base model, not to the authors. Our speech model is initialized from MoshiRAG (Chien et al., 2026), which the paper cites as third-party work. Naming a system after the open model it builds on is common practice (e.g., LLaMA-Omni, WhisperX) and does not indicate affiliation.

Q3. Types of tool calls and Python execution

The tool calls follow the standard function-calling format of current foundation models, in which a named function with typed arguments returns a result. The data come from nine tool-use sources (Table 13), the largest being Toucan (42.3%) and Nemotron-SFT Agentic (31.7%), and cover customer service, retrieval, function calling, and multi-step agent interaction. The speech model does not emit tool calls. The reasoning backend issues them through standard function calling, so any client-executed function with a JSON schema, including a Python-execution tool, can in principle be exposed. We added this description to Sec. 4.

Thank you for your response. It is great work and I enjoyed the read. There is a couple of things I don't quite get.

  1. Like you mention the 184 ms smooth-turn and 388 ms interruption latency, which is great, but that only reflects how quickly the speaker responds instead of how fast a task actually finishes. My concern is still that adding extra checks and balances delays what the user wants, which might not always be helpful. Taking the whole pipeline into account (including reasoning and tool execution), how long does a full task normally take from start to finish in absolute terms? I think its could be a drawback that is slows down normal tasks.
  2. One other thing, in Table 3, why does the Revision-bound execution drop in performance on FSB-V3 and stays about the same for t-Voice for Moshi-RAG but not for MoshiAgent-Speech?
  3. I think in general its a bit confusing that you say the confirmation is not always required yet one of the core components of your work is built around it

So I maintain the rating of Weak Acceptance.

Reviewer_PibROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies state consistency in full-duplex voice agents, where correctly executed tool operations may no longer match the user’s revised intent. The authors propose revision-bound grounding, which enforces four session-local properties to keep asynchronous evidence and actions aligned with evolving user intent. They instantiate this design in MoshiAgent and construct ToolRaceBench from τ-Voice to evaluate the targeted scenarios. Starting from MoshiRAG, they train MoshiAgent-Speech on 599,411 conversations totaling 12,106.7 hours. With the trained speech model fixed, revision-bound execution substantially improves performance over a direct tool loop on ToolRaceBench and also yields gains on existing tool-use and task-completion benchmarks.

Critical strengths

S1. Important problem and well-motivated state distinctions

The paper addresses a practically important consistency problem: ongoing reasoning and tool execution can lose validity as user intent changes. The distinction between dialogue state, action state, and observed world state is well motivated. In particular, distinguishing superseded pending actions from external effects that have already occurred provides a useful basis for handling user revisions.

S2. Informative experiments and targeted evaluation

Table 3 crosses speech models with execution policies, while Table 4 provides component ablations. These experiments offer useful evidence for the contributions of speech training and the execution framework. ToolRaceBench complements existing end-to-end benchmarks with controlled scenarios and behavioral oracles for result attribution, outcome reuse, authorization binding, and user revision.

Critical weaknesses

W1. Under-specified execution baseline

Although Table 3 fixes the reasoning backend, tool schemas, and evaluation inputs, the implementation differences between the Direct Tool Loop and revision-bound execution remain insufficiently specified. It is unclear how the policies differ in history retention, access to previous tool results, reasoning coordination, and retry or budget settings. While Table 4 provides useful component ablations, these missing details limit precise attribution of the gains to the controls in Section 3.3.

W2. Insufficient operational specification

The paper defines useful interface properties and describes revision identifiers and pre-dispatch freshness checks, but it does not provide a sufficiently explicit execution procedure. The operational rules for canonical action equivalence, result applicability, and semantic supersession remain unclear, as does the synchronization between freshness validation and dispatch under concurrent user revisions. These details are necessary to reproduce the proposed runtime’s behavior.

Questions for authors

Q1. Definition of the execution comparison

Could the authors specify the implementation differences between the Direct Tool Loop and revision-bound execution, particularly regarding history retention, reasoning coordination, and retry or budget settings? Which comparisons in Tables 3 and 4 most directly isolate the controls proposed in Section 3.3 from other changes to the runtime?

Q2. Operational specification of the runtime

Could the authors provide pseudocode or an explicit state-transition procedure for the revision-bound runtime? In particular, how are action equivalence, result applicability, and semantic supersession determined, and how is freshness validation synchronized with dispatch when user revisions arrive concurrently?

1 comment
AuthorsOpenReview ↗

Response to Reviewer PibR

We thank the reviewer for the careful reading and constructive suggestions.

W1 & Q1. Differences between the two execution policies, and which comparisons isolate the Sec. 3.3 controls

We agree that the policies were under-specified. Sec. 5.2 and Table 19 (Appendix D) now describe them. Both use the same speakers, reasoning backend (Gemma-4-31B-it), tool schemas, and evaluation inputs.

Direct Tool LoopRevision-bound execution
History across turnsRetainedRetained
Access to previous tool resultsYesYes
Retry and budget settingsSameSame
Reasoning coordinationNoneCoordinator with Fast Reasoner and Deliberative Explorer
Write controlNone (writes executed directly)Exact-proposal binding (P3) and freshness check before dispatch (P4)
Result replayNoneObserved successes only (P2)
Evidence admissionNoneCorrelation and freshness checks (P1, P4)

Since the policies differ in both the grounding controls and the reasoning coordination, Table 3 reflects their joint effect, as Sec. 5.4 now states. The most direct comparisons are the single-component removals in Table 4. The two rows that remove a grounding control keep everything else unchanged, including Reasoner History and coordination.

Removed componentControls removedToolRaceBenchτ-Voice
None (full system)83.232.1
Evidence AdmissionP1, P4 on returned evidence42.6 (−40.6)27.4 (−4.7)
Revision-Bound Write ControlP3, pre-dispatch P450.7 (−32.6)26.3 (−5.8)

These drops can therefore be attributed to the removed controls.

W2 & Q2. Operational specification of the runtime

Appendix A now gives the state schema (Listing 1), pseudocode for proposal resolution, guarded dispatch, evidence admission, replay, and scheduling (Listings 2 to 6), and an end-to-end trace (Table 11). The key rules, also stated in Secs. 3.2 and 3.3, are as follows.

Action equivalence. Two calls are equivalent if they have the same canonical tool name and canonicalized arguments. Equivalence is used only to reuse a read result. It never suppresses or authorizes a write.

Result applicability (P2). A stored outcome is applicable while no relevant write has occurred since it was observed. If this cannot be established, it is treated as not applicable. An applicable outcome is reused only if it is a correlated success and the requesting intent is still fresh. Otherwise the tool is called again (Listing 5).

Semantic supersession. An operation is superseded when later user speech withdraws its intent or changes an action-relevant target, argument, or scope. Compatible speech does not supersede it (Sec. 3.2). For a pending proposal, the subsequent speech is resolved against the frozen action CpC_p as accept, reject, modify, or unclear. Deterministic rules decide clear consent or refusal (e.g., a bare "yes" or "no"). Speech with a modification cue or mixed affirmative and negative words goes to the semantic classifier, which is the reasoning backend itself (Gemma-4-31B-it by default). All other speech (e.g., a question) is unclear and never grants authorization. A modify decision supersedes the proposal and triggers replanning. A classifier verdict is committed only if the proposal and the revision are unchanged since its snapshot (Listing 2).

Synchronization of freshness validation and dispatch (Appendix A.3). All state changes in a session, including the commit of new user speech, are applied one at a time as atomic steps. Dispatch is one such step (Listing 3, condensed below), so no revision can be committed between the freshness check and the send.

DISPATCH(p)                                                  # one atomic step
  if not authorized_exactly(p.pid, p.C_p) then return NOT_AUTHORIZED     # P3
  if freshness(p, D_t) != TRUE then return BLOCKED                       # P4, unresolved does not pass
  if dispatch_started(p) then return DUPLICATE_DISPATCH
  register the invocation and send exactly p.C_p

A racing revision is therefore handled in one of two ways. If it is committed first, freshness is unresolved and dispatch is blocked. A later reject or modify then prevents the dispatch. If dispatch runs first, the verified effect of the dispatched action is recorded in the world state but cannot satisfy the revised request or support a completion claim, and the agent replans.

We hope these additions address the reviewer's concerns and would be grateful if the reviewer would consider updating the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.