- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether an assistant can recover a pending task after intervening conversation, respond when the necessary information first becomes complete, and identify missing information when it remains incomplete. PTE-BENCH contains 2,268 conversations and 10,639 evaluation instances, separating active recall at the first complete prefix from state recognition at earlier prefixes. ARDSE maintains tasks, operations, requirements, and evidence across conversation blocks, selects a task using relevance, recent state changes, and continuity, then verifies requirements before generating a response. Across four backbones and two task-information conditions, it leads all 16 primary accuracy comparisons against the reported baselines. The experiments assess text responses at selected prefixes of constructed conversations, without verifying real tool execution or continuous online triggering.
Critical strengths
- The problem is relevant to long-running assistants. Remembering a pending task when its information becomes sufficient and withholding action before that point are distinct capabilities. Evaluating both, while separating exact instance correctness from field coverage, avoids treating partial retrieval as successful task completion.
- The benchmark construction uses explicit state and evidence constraints. Seed conversations are divided into stages, expanded with exchanges that should not advance the target task, checked for terminal completeness, and annotated with minimal supporting utterances. Four scenarios and two initial-information conditions provide a clear testing structure.
- The system-level improvements are consistent across backbones. ARDSE leads all 16 ARAcc and StateAcc comparisons; for example, GPT-4o Mini With Task improves from 39.68% to 61.11% ARAcc over Full-Context. The full method also leads the ARAcc and RIC ablations, supporting the usefulness of the combined state-maintenance and verification procedure.
- The appendices retain details that help interpret the results, including poor cross-session performance, sample counts for graph-size groups, treatment of constant activation signals, and the assumptions behind the estimated speedup. These disclosures make the remaining difficulties visible.
Critical weaknesses
- Required information and earliest executability need a clearer operational basis. The prompts exclude optional preferences, yet examples retain cuisine, area, or price as required information after a restaurant has been selected. These may be legitimate user constraints; the excerpts do not establish annotation errors. The paper should distinguish operation parameters, explicit user requirements, and inherited source-dataset slots. Human checks also lack sample counts, disagreement, and correction rates, making the reliability of strict-set scoring difficult to assess.
- The implemented pipeline and the independent contribution of graph structure remain unclear. Global reconciliation in the appendix rereads the full conversation, and verification and generation can access it again. The main comparison therefore primarily establishes the effectiveness of the complete multi-stage system. The semantic encoder, graph integration, and activation tie-breaking are underspecified, as are the relationships, reconciliation steps, and call arrangements retained in flat-state and other ablations. This limits attribution to graph organization rather than additional full-context reasoning.
- Reliability claims should better reflect difficult settings and adverse ablations. Without Task, StateAcc is only 3.74%–11.27%, while cross-session ARAcc is 2.33%–5.00%. For Qwen With Task, removing verification raises StateAcc from 34.72% to 43.93% while reducing ARAcc by only .44 percentage points. The constructed trajectories mainly resolve tentative information, and all conditional trajectories eventually satisfy their trigger. The results support meaningful relative improvements, but not a general claim that verification benefits every backbone or that broadly reliable, timely execution has been established.
Questions for authors
- What specification determines the minimum required information? Using the existing examples in Tables 5 and 8, please explain when cuisine, area, or price remains essential after the restaurant is selected, and how stage boundaries map to the earliest executable turn.
- How many conversations, prefixes, and judge outputs were checked by humans, how were subsets sampled, and what disagreements or corrections appear in the existing records?
- Where does global reconciliation occur in the implemented call sequence, and which semantic encoder and tie-breaking rule are used? What full-context processing and calls remain in the flat-state, independent-block, and no-verification variants?
- Do existing outputs explain why removing verification improves Qwen StateAcc by 9.21 points while reducing ARAcc by only .44 points? How should this qualify the reliability claim for verification?
5 comments
Rebuttal to Reviewer 45uG (continue)
Q1: Required fields and first executability. See W1, Operational basis and Examples in Tables 5 and 8.
Q2: Annotation and judge checks. See W1, Human checks.
Q3: Reconciliation, encoder, and retained calls. With Task State Recognition reconciles after block extraction, reading the full prefix and mapping a provisional answer to fields in two calls. Without Task State Recognition grounds requirements in the prefix. Flat state retains the applicable full-prefix calls. Independent blocks merge first, with reconciliation also in With Task Active Recall. The no-verification variant generates from provisional state and the full prefix. Activation uses cosine similarity over lowercase token counts for the last six turns and task description. Ties favor open tasks, then the latest relevant turn, then ledger order.
Q4: Qwen's two accuracy measures. See W3, Qwen With Task, for the explanation and the concert and restaurant example.
Thank you again for the constructive suggestions.
Dear Reviewer 45uG,
Thank you again for your detailed and constructive review. As the discussion period draws to a close, we would be grateful if you had a chance to consider our rebuttal. We wanted to highlight the points most directly related to your assessment.
-
Required information and first executability. Tables 5 and 8 illustrate how we ground the requirements in the source dialogue. In Table 5, cuisine, area, and price are user-stated venue-selection criteria. In Table 8, the user explicitly asks about price range in the full dialogue, and the illustrated prefix still lacks the date, time, and party size. We also summarized the human annotation details, which covered approximately 50% of the benchmark and yielded a pre-discussion agreement of κ = 0.74.
-
Graph structure. We clarified which fields and inference steps remain in the flat-state ablation. Both variants can access the same full prefix and use the applicable downstream prompts. In the reported Without Task comparison, full ARDSE leads the flat variant by an average of 5.60 percentage points in ARAcc and 10.47 points in RIC. This comparison addresses the representation of maintained task state within the pipeline.
-
Reliability and the Qwen ablation. We discussed what the low Without Task and Cross-Session scores reveal about recovering a pending task across intervening dialogue. We also examined a Qwen case where reconciliation assigned a concert’s start time and ticket count to a restaurant reservation. Removing that step preserved the correct missing fields in this case. This illustrates why StateAcc can improve on incomplete prefixes while reported ARAcc changes little.
We would especially value your view on whether these clarifications address your concerns about benchmark validity and the interpretation of the ablations. If a specific concern still affects your assessment, we would welcome the opportunity to address it. If the discussion changes any part of your assessment, we would appreciate an update to your review.
Thank you again for your time and careful consideration.
Rebuttal to Reviewer 45uG
Dear Reviewer 45uG,
Thank you for the constructive review and for highlighting the distinction between task recall and readiness.
W1: Required information and first executability.
Operational basis. We base both the required information and the first executable point on the original task and its supporting dialogue evidence. SGD and MultiWOZ provide service schemas and dialogue-state annotations that anchor the operation and its candidate arguments. We retain fields needed to carry out that operation or satisfy an explicit user constraint or request. For each field, we annotate the stage at which the dialogue establishes its value for that specific task. The first executable prefix is the earliest one in which all required values are established (§4, Algorithm 1).
Examples in Tables 5 and 8. In Table 5, selecting Saffron Brasserie satisfies the cuisine, area, and price criteria. The booking becomes executable when the user confirms the time at stage 5. In Table 8, the user explicitly requests price range in the full dialogue. Date, time, and party size remain unresolved at the illustrated prefix.
Human checks. The annotation audit covered about 50% of the benchmark. Pre-discussion agreement was κ = 0.74. After consensus adjudication, about 10% of reviewed cases required corrections to AI-generated annotations. We separately spot-checked responses and GPT-4o Mini judgments (Appendix D). We will report exact sample counts and sampling procedures in the revision.
W2: Graph structure and full-prefix reasoning.
Why a task-state graph. A pending task connects an operation to its requirements, their current values and statuses, and the turns that support them. As §5 describes, “The edges encode their semantic, evidential, and temporal relations.” For example, REQUIRES links an operation to its requirements, SUPPORTS links a value to its evidence, and UPDATES records revisions. These links let ARDSE apply new evidence to the corresponding requirement and carry unresolved task state across topic shifts (Fig. 2 and Eq. 1).
Why task activation. Several requests can remain pending across topic shifts, and a new detail can advance an earlier request during an unrelated exchange. We designed activation to identify which stored task the new evidence makes relevant, so the assistant can resume it when ready without the user restating it (§5, Eq. 2).
Flat-state ablation. w/o Graph retains task and evidence IDs, applicable downstream prompts, and the full prefix. The Without Task differences are Full ARDSE − w/o Graph, in percentage points (Table 3).
Both reported metrics favor full ARDSE across backbones. The activation ablation selects the most recently mentioned task (§6.3).
W3: Interpreting the accuracy results.
Without Task. Removing the task specification lowers ARAcc by 22.79–39.68 and StateAcc by 30.98–45.60 across all four backbones (Table 2). The ARAcc/StateAcc drops are 39.68/45.60 for GPT-4o Mini, 39.33/43.75 for Claude, 22.79/36.01 for Llama, and 39.02/30.98 for Qwen. Without Task StateAcc is 3.74–11.27%. Cross-Session, where evidence is spread across conversations, has the lowest Without Task ARAcc on every backbone (2.33–5.00%, Table 6). The declines are consistent with the preliminary study, where removing the task reminder from the same dialogues reduced mean execution accuracy from 30.0% to 13.0% (§3, Fig. 1b). When the task is unstated, the model must recover the intended request before assessing its fields.
Qwen With Task. With the schema supplied, the initial state can correctly flag missing fields. Reconciliation can overwrite that status when it extracts a plausible value for the wrong task. In one dialogue, the reconciled state assigned a concert's 8:30 p.m. start time and two-ticket count to a restaurant's reservation time and party size. The trace suggests that the pass matched the time and count by field type and overlooked which request they described. Qwen indicated readiness. Without the pass, Qwen correctly reported both reservation fields as missing. This illustrates how StateAcc on incomplete prefixes can rise (34.72% → 43.93%) while ARAcc on ready prefixes changes little.