acceptodds
Under review as a conference paper at ICLR 2027

Proactive Recall for LLMs: Towards Reliable and Timely Task Execution

Abstract

Recent advances in LLM-based dialogue systems have substantially improved their ability in task execution. However, lacking the ability to proactively recall necessary information for pending tasks, they misjudge whether to execute: acting with fabricated values when information is insufficient or staying inactive despite sufficient information already being available in the interaction history. To address this limitation, we formalize Proactive Task Execution (PTE) and introduce PTE-Bench, a benchmark for trustworthy and timely LLM task execution. PTE-Bench encompasses 2,268 long conversations and 10,639 evaluation instances across four conversation scenarios, providing a comprehensive testbed for probing whether an LLM can proactively recall task-relevant information to decide whether to execute, and to identify what is missing when it cannot. We further propose ARDSE, Active Recall through Dynamic State Evolution, which supports proactive recall by dynamically monitoring pending task states through a Progressive Task-State Graph and using evidence-conditioned activation to determine which task to recall in multi-task conversations. Extensive experiments on four LLM backbones and five representative baselines show that ARDSE achieves the best performance, demonstrating the effectiveness of persistent task-state evolution for proactive task execution. Data is available at https://anonymous.4open.science/r/active_recall-65ED.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Proactive Recall, LLM, LLM memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_45uGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether an assistant can recover a pending task after intervening conversation, respond when the necessary information first becomes complete, and identify missing information when it remains incomplete. PTE-BENCH contains 2,268 conversations and 10,639 evaluation instances, separating active recall at the first complete prefix from state recognition at earlier prefixes. ARDSE maintains tasks, operations, requirements, and evidence across conversation blocks, selects a task using relevance, recent state changes, and continuity, then verifies requirements before generating a response. Across four backbones and two task-information conditions, it leads all 16 primary accuracy comparisons against the reported baselines. The experiments assess text responses at selected prefixes of constructed conversations, without verifying real tool execution or continuous online triggering.

Critical strengths
  • The problem is relevant to long-running assistants. Remembering a pending task when its information becomes sufficient and withholding action before that point are distinct capabilities. Evaluating both, while separating exact instance correctness from field coverage, avoids treating partial retrieval as successful task completion.
  • The benchmark construction uses explicit state and evidence constraints. Seed conversations are divided into stages, expanded with exchanges that should not advance the target task, checked for terminal completeness, and annotated with minimal supporting utterances. Four scenarios and two initial-information conditions provide a clear testing structure.
  • The system-level improvements are consistent across backbones. ARDSE leads all 16 ARAcc and StateAcc comparisons; for example, GPT-4o Mini With Task improves from 39.68% to 61.11% ARAcc over Full-Context. The full method also leads the ARAcc and RIC ablations, supporting the usefulness of the combined state-maintenance and verification procedure.
  • The appendices retain details that help interpret the results, including poor cross-session performance, sample counts for graph-size groups, treatment of constant activation signals, and the assumptions behind the estimated speedup. These disclosures make the remaining difficulties visible.
Critical weaknesses
  • Required information and earliest executability need a clearer operational basis. The prompts exclude optional preferences, yet examples retain cuisine, area, or price as required information after a restaurant has been selected. These may be legitimate user constraints; the excerpts do not establish annotation errors. The paper should distinguish operation parameters, explicit user requirements, and inherited source-dataset slots. Human checks also lack sample counts, disagreement, and correction rates, making the reliability of strict-set scoring difficult to assess.
  • The implemented pipeline and the independent contribution of graph structure remain unclear. Global reconciliation in the appendix rereads the full conversation, and verification and generation can access it again. The main comparison therefore primarily establishes the effectiveness of the complete multi-stage system. The semantic encoder, graph integration, and activation tie-breaking are underspecified, as are the relationships, reconciliation steps, and call arrangements retained in flat-state and other ablations. This limits attribution to graph organization rather than additional full-context reasoning.
  • Reliability claims should better reflect difficult settings and adverse ablations. Without Task, StateAcc is only 3.74%–11.27%, while cross-session ARAcc is 2.33%–5.00%. For Qwen With Task, removing verification raises StateAcc from 34.72% to 43.93% while reducing ARAcc by only .44 percentage points. The constructed trajectories mainly resolve tentative information, and all conditional trajectories eventually satisfy their trigger. The results support meaningful relative improvements, but not a general claim that verification benefits every backbone or that broadly reliable, timely execution has been established.
Questions for authors
  • What specification determines the minimum required information? Using the existing examples in Tables 5 and 8, please explain when cuisine, area, or price remains essential after the restaurant is selected, and how stage boundaries map to the earliest executable turn.
  • How many conversations, prefixes, and judge outputs were checked by humans, how were subsets sampled, and what disagreements or corrections appear in the existing records?
  • Where does global reconciliation occur in the implemented call sequence, and which semantic encoder and tie-breaking rule are used? What full-context processing and calls remain in the flat-state, independent-block, and no-verification variants?
  • Do existing outputs explain why removing verification improves Qwen StateAcc by 9.21 points while reducing ARAcc by only .44 points? How should this qualify the reliability claim for verification?
5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 45uG

Dear Reviewer 45uG,

Thank you for the constructive review and for highlighting the distinction between task recall and readiness.

W1: Required information and first executability.

Operational basis. We base both the required information and the first executable point on the original task and its supporting dialogue evidence. SGD and MultiWOZ provide service schemas and dialogue-state annotations that anchor the operation and its candidate arguments. We retain fields needed to carry out that operation or satisfy an explicit user constraint or request. For each field, we annotate the stage at which the dialogue establishes its value for that specific task. The first executable prefix is the earliest one in which all required values are established (§4, Algorithm 1).

Examples in Tables 5 and 8. In Table 5, selecting Saffron Brasserie satisfies the cuisine, area, and price criteria. The booking becomes executable when the user confirms the time at stage 5. In Table 8, the user explicitly requests price range in the full dialogue. Date, time, and party size remain unresolved at the illustrated prefix.

Human checks. The annotation audit covered about 50% of the benchmark. Pre-discussion agreement was κ = 0.74. After consensus adjudication, about 10% of reviewed cases required corrections to AI-generated annotations. We separately spot-checked responses and GPT-4o Mini judgments (Appendix D). We will report exact sample counts and sampling procedures in the revision.

W2: Graph structure and full-prefix reasoning.

Why a task-state graph. A pending task connects an operation to its requirements, their current values and statuses, and the turns that support them. As §5 describes, “The edges encode their semantic, evidential, and temporal relations.” For example, REQUIRES links an operation to its requirements, SUPPORTS links a value to its evidence, and UPDATES records revisions. These links let ARDSE apply new evidence to the corresponding requirement and carry unresolved task state across topic shifts (Fig. 2 and Eq. 1).

Why task activation. Several requests can remain pending across topic shifts, and a new detail can advance an earlier request during an unrelated exchange. We designed activation to identify which stored task the new evidence makes relevant, so the assistant can resume it when ready without the user restating it (§5, Eq. 2).

Flat-state ablation. w/o Graph retains task and evidence IDs, applicable downstream prompts, and the full prefix. The Without Task differences are Full ARDSE − w/o Graph, in percentage points (Table 3).

BackboneARAccRIC
GPT-4o Mini+4.72+8.62
Claude+10.14+15.88
Llama+4.99+10.57
Qwen+2.55+6.82
Mean+5.60+10.47

Both reported metrics favor full ARDSE across backbones. The activation ablation selects the most recently mentioned task (§6.3).

W3: Interpreting the accuracy results.

Without Task. Removing the task specification lowers ARAcc by 22.79–39.68 and StateAcc by 30.98–45.60 across all four backbones (Table 2). The ARAcc/StateAcc drops are 39.68/45.60 for GPT-4o Mini, 39.33/43.75 for Claude, 22.79/36.01 for Llama, and 39.02/30.98 for Qwen. Without Task StateAcc is 3.74–11.27%. Cross-Session, where evidence is spread across conversations, has the lowest Without Task ARAcc on every backbone (2.33–5.00%, Table 6). The declines are consistent with the preliminary study, where removing the task reminder from the same dialogues reduced mean execution accuracy from 30.0% to 13.0% (§3, Fig. 1b). When the task is unstated, the model must recover the intended request before assessing its fields.

Qwen With Task. With the schema supplied, the initial state can correctly flag missing fields. Reconciliation can overwrite that status when it extracts a plausible value for the wrong task. In one dialogue, the reconciled state assigned a concert's 8:30 p.m. start time and two-ticket count to a restaurant's reservation time and party size. The trace suggests that the pass matched the time and count by field type and overlooked which request they described. Qwen indicated readiness. Without the pass, Qwen correctly reported both reservation fields as missing. This illustrates how StateAcc on incomplete prefixes can rise (34.72% → 43.93%) while ARAcc on ready prefixes changes little.

AuthorsOpenReview ↗

Rebuttal to Reviewer 45uG (continue)

Q1: Required fields and first executability. See W1, Operational basis and Examples in Tables 5 and 8.

Q2: Annotation and judge checks. See W1, Human checks.

Q3: Reconciliation, encoder, and retained calls. With Task State Recognition reconciles after block extraction, reading the full prefix and mapping a provisional answer to fields in two calls. Without Task State Recognition grounds requirements in the prefix. Flat state retains the applicable full-prefix calls. Independent blocks merge first, with reconciliation also in With Task Active Recall. The no-verification variant generates from provisional state and the full prefix. Activation uses cosine similarity over lowercase token counts for the last six turns and task description. Ties favor open tasks, then the latest relevant turn, then ledger order.

Q4: Qwen's two accuracy measures. See W3, Qwen With Task, for the explanation and the concert and restaurant example.

Thank you again for the constructive suggestions.

Dear Reviewer 45uG,

Thank you again for your detailed and constructive review. As the discussion period draws to a close, we would be grateful if you had a chance to consider our rebuttal. We wanted to highlight the points most directly related to your assessment.

  • Required information and first executability. Tables 5 and 8 illustrate how we ground the requirements in the source dialogue. In Table 5, cuisine, area, and price are user-stated venue-selection criteria. In Table 8, the user explicitly asks about price range in the full dialogue, and the illustrated prefix still lacks the date, time, and party size. We also summarized the human annotation details, which covered approximately 50% of the benchmark and yielded a pre-discussion agreement of κ = 0.74.

  • Graph structure. We clarified which fields and inference steps remain in the flat-state ablation. Both variants can access the same full prefix and use the applicable downstream prompts. In the reported Without Task comparison, full ARDSE leads the flat variant by an average of 5.60 percentage points in ARAcc and 10.47 points in RIC. This comparison addresses the representation of maintained task state within the pipeline.

  • Reliability and the Qwen ablation. We discussed what the low Without Task and Cross-Session scores reveal about recovering a pending task across intervening dialogue. We also examined a Qwen case where reconciliation assigned a concert’s start time and ticket count to a restaurant reservation. Removing that step preserved the correct missing fields in this case. This illustrates why StateAcc can improve on incomplete prefixes while reported ARAcc changes little.

We would especially value your view on whether these clarifications address your concerns about benchmark validity and the interpretation of the ablations. If a specific concern still affects your assessment, we would welcome the opportunity to address it. If the discussion changes any part of your assessment, we would appreciate an update to your review.

Thank you again for your time and careful consideration.

Reviewer_Ru6LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates proactive task recall and timely execution in long conversations, introducing PTE-Bench to evaluate task-state recognition and active recall, along with ARDSE, a framework that maintains evolving task states and activates tasks when their execution conditions are satisfied. Experiments on PTE-Bench demonstrate improvements over five baseline methods, while ablation studies support the effectiveness of its key components.

Critical strengths
  1. The paper addresses proactive task recall and timely execution, an important yet underexplored challenge for LLM-based assistants.
  2. PTE-Bench provides a structured evaluation framework for assessing task-state recognition and active recall in long conversations.
  3. ARDSE introduces a progressive task-state graph to track evolving task requirements and reactivate tasks when execution conditions are satisfied.
  4. Extensive comparisons with five baselines and ablation studies demonstrate the effectiveness of the proposed framework and its key components.
Critical weaknesses
  1. The benchmark construction pipeline involves eight different prompts, but the paper lacks an end-to-end example demonstrating how these prompts are applied to a complete SGD or MultiWOZ dialogue. Providing concrete examples would significantly improve clarity and reproducibility.
  2. Several mathematical symbols and variables are introduced without adequate definitions or explanations, making the methodology difficult to follow and potentially causing ambiguity.
  3. The benchmark relies heavily on task-state evolution, yet the segmentation process is primarily guided by natural-language prompts without explicit or objective criteria. The authors should explain why existing dialogue-state annotations are not directly used to identify state transitions.
  4. Since task-progress segmentation depends on LLM judgments, different models may produce inconsistent segment boundaries. The paper lacks sufficient analysis of segmentation consistency, reliability, and the potential impact of segmentation variability on benchmark quality.
  5. The paper largely equates task executability with the completeness of required information, overlooking whether the user still intends or authorizes the task to be executed.
  6. By enforcing that tasks become executable only at the final stage, PTE-Bench introduces an artificial task progression pattern that may not reflect realistic conversational dynamics.
Questions for authors
  1. Given that the entire benchmark fundamentally relies on task-state evolution, why do the authors fail to clearly explain how stage boundaries are reliably determined, instead relying primarily on a natural-language prompt? Why not directly use changes in dialogue-state annotations to identify segmentation boundaries? Furthermore, how can the consistency of segmentation results across different LLMs be ensured under the proposed approach?
  2. The benchmark construction process ensures that all required information becomes available only at the final task stage. Could this artificial constraint introduce predictable patterns that models exploit, rather than genuinely evaluating their proactive recall capabilities?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Ru6L

Dear Reviewer Ru6L,

Thank you for the constructive review and for recognizing the importance of proactive task recall.

W1: Construction example. Table 5 traces a MultiWOZ reservation from a proposed time in stage 4 to its confirmation in stage 5. The anonymous release linked in the abstract provides the construction code and prompts. Given the space limit, we will include a complete worked example with intermediate inputs and outputs in the revision.

The table follows the same reservation from the MultiWOZ seed dialogue to the benchmark instances. Each row shows the output passed to the next step.

StepOutput for the Table 5 example
Task extraction (Table 11)Identify the reservation and seven required fields: cuisine, price range, area, venue, day, time, and party size.
Segmentation (Table 12)Group contiguous turns by task progress, from stating venue criteria to selecting a restaurant and confirming the booking details.
Rewriting (Table 13)Render the stages as natural user–assistant exchanges while preserving the request, values, and task progression.
State annotation (Table 14)Map Indian cuisine and an expensive price range to stage 1, the town centre to stage 2, and Saturday, six guests, and Saffron Brasserie to stage 4. Stage 3 establishes no new field. The user confirms 19:30 in stage 5.
Validation and repair if needed (Table 18)Check that every required value is supported and identify the first complete stage. Repair invalid trajectories and validate them again before expansion.
Topic proposal (Table 15)Propose off-task topics for the gaps between stages.
Off-task expansion (Table 16)Insert those exchanges while preserving the reservation state, including the unresolved booking time before stage 5.
Evidence annotation (Table 17)Record the minimal supporting turns for each value, such as the user's explicit selection of Saffron Brasserie in stage 4.

The final records use earlier incomplete prefixes for Task State Recognition, each labeled with its outstanding requirements. The first complete prefix, in stage 5, is the Active Recall instance with the target operation and all required values (Algorithm 1).

W2: Notation. Thank you for flagging this. We will define symbols at first use, including o_i (operation), R_i (required field–value pairs), and m_i* (first complete stage).

W3/W4/Q1: Segmentation and quality control. We appreciate the insightful question. Source annotations track dialogue state. PTE-BENCH additionally labels when evidence establishes each required value for a pending task. With the full seed dialogue and required fields, the prompt groups turns by where values are introduced, revised, or confirmed (Table 12). Several LLMs produced similar segmentations in our pilot. We chose GPT-4o Mini for the full annotation pass to control cost and apply a consistent procedure. Intermediate turns can admit multiple valid groupings. We check stage contiguity, field coverage, and whether the first complete stage agrees with the dialogue evidence. Three NLP researchers also check the quality.

W5: Readiness and authorization. We appreciate the distinction. PTE-BENCH evaluates whether the assistant recognizes and reports readiness, including satisfaction of any user-stated condition. A deployed assistant can use that readiness decision to request confirmation before invoking a tool when privacy or authorization calls for it.

W6/Q2: Realism and positional cues. Thank you for raising this concern. Long dialogues in which information for a pending task arrives over many turns are difficult to collect. Our seed data, SGD and MultiWOZ, model realistic problems and practical requests such as restaurant reservations, and they are high cited and have high quality. We retain the source task progression and insert state-preserving off-task exchanges. This lets us test whether an assistant can recognize when enough information has accumulated to resume a pending request after intervening dialogue.

Thank you again for the constructive suggestions.

Dear Reviewer Ru6L,

Thank you again for your constructive feedback. Thank you again for your thoughtful review. As the discussion period draws to a close, we would appreciate it if you had a chance to look at our response. We added a step-by-step walkthrough of the example in Table 5 and clarified how task segmentation is conducted and checked. We also addressed your questions about user authorization and whether the final-stage design offers a positional shortcut.

We would be happy to clarify any points that remain open. If these explanations change your assessment, we would appreciate an update to your review. Thank you again for helping us improve the paper.

Reviewer_GxsAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a conversational agent can resume a pending task when its requirements become available, and identify missing information when they do not. PTE-Bench contains 2,268 conversations and 10,639 evaluation instances derived from task-oriented dialogues and synthetic cross-session and conditional scenarios. ARDSE maintains a task-state graph, activates relevant pending tasks, and verifies their requirements before responding. Experiments cover four backbones, five baselines, and component ablations.

Critical strengths

The distinction between retaining information and resuming an executable intention is useful. Evaluating both complete and incomplete task states makes the benchmark more informative than testing recall alone. The construction also preserves evidence and task-state annotations, which can support diagnosis of errors.

ARDSE improves active-recall accuracy across all four backbones in both settings. For example, GPT-4o Mini improves from 39.68 to 61.11 with the task specification and from 4.63 to 21.43 without it (Table 2). The component ablations support the value of maintaining and verifying task state. These results and the benchmark contribution make me lean toward acceptance, despite the evaluation limitations below.

Critical weaknesses
  1. The comparisons do not fully isolate the value of the graph and activation mechanism from task-specific prompting and additional inference. The full-context Active Recall prompt in Table 23 is very general, while ARDSE receives detailed instructions for requirement extraction, reconciliation, and evidence verification (Tables 19–22). Appendix H compares efficiency with A-MEM, but does not establish whether a simpler structured task ledger with comparable instructions and inference budget would perform similarly. This limits the algorithmic conclusion, although it does not remove the benchmark's value.

  2. The evidence supports a narrower notion of reliable and timely execution than the framing suggests. Evaluation uses selected conversation prefixes and generated responses, rather than letting the agent's actions change subsequent interaction. Appendix C also states that confirmed values are not subsequently replaced and that every conditional trajectory eventually triggers. The setup therefore does not establish robustness to cancelled intentions, revised commitments, or never-triggered conditions. This matters given the 3.74–11.27 StateAcc without task specifications in Table 2. The annotation and judge checks in Appendix D also lack sample counts and quantitative agreement results.

Questions for authors
  1. How does a flat task ledger perform when given the same requirement-extraction and verification instructions and a comparable inference budget? The existing graph ablation is helpful; please clarify which instructions, state fields, and calls it preserves.
  2. How many annotations and model judgments were independently checked, how were they sampled, and what were the agreement and correction rates?
  3. Can the current results separate premature execution from incorrect missing-field identification? Please clarify whether any evaluation measures first-action timing across successive prefixes, including off-task turns.
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer GxsA

Dear Reviewer GxsA,

Thank you for the constructive review and for recognizing the distinction between retaining information and resuming a pending task.

W1/Q1: Baseline prompts. Full-Context Prompting responds directly to the dialogue prefix. ARDSE's prompts build and check task state from the same dialogue (Tables 19–22). Table 23 specifies readiness and output format for State Recognition, as does the With Task implementation. We will make these instructions explicit in the general Active Recall prompt.

Flat ledger. Table 3's w/o Graph variant flattens the selected state, retaining task IDs, fields, values, status, and supporting turns. It keeps the extraction, applicable verification, and generation prompts and the same prefix, therefore have similar efficency. Each cell gives Full ARDSE − w/o Graph in percentage points, in ARAcc/RIC/StateAcc/MIC order (Table 3). The results demonstrates an overall improvement.

BackboneWith TaskWithout Task
GPT-4o Mini+6.88/+6.77/+1.15/+0.78+4.72/+8.62/+0.22/+0.11
Claude+5.77/+7.01/+3.20/+2.54+10.14/+15.88/+2.05/−0.85
Llama+5.29/+5.99/+0.11/+0.18+4.99/+10.57/+0.09/+0.57
Qwen+0.35/+0.22/−8.27/−7.54+2.55/+6.82/+0.78/+5.10

The ARAcc and RIC gains in both settings support the contribution of structured state. With Task Active Recall has matching block-update and reconciliation-pass counts.

W2/Q2: Dialogue readiness. Our primary setting is conversation. Recognizing readiness and reporting the correct task state are prerequisites for agent action. For a reservation, the assistant should identify missing details while the request is incomplete and provide the correct booking details once it is ready. We appreciate the suggestion to study how an executed action changes subsequent interaction.

Task-state trajectories. We preserve the source progression from tentative alternatives to confirmed requirements, and confirmed values remain stable for the same request (Appendix C). If–Then evaluates the request before and when its trigger is satisfied.

Human checks. The annotation audit covered about 50% of the benchmark. Pre-discussion agreement was κ = 0.74. After consensus adjudication, about 10% of reviewed cases required corrections to AI-generated annotations. We separately spot-checked responses and GPT-4o Mini judgments (Appendix D). We will report exact sample counts and sampling procedures in the revision.

Q3: Error types and timing. At an incomplete prefix, attempting execution, claiming completion, or committing to execution unconditionally is premature. A response that withholds action has a missing-field error if it omits an unresolved requirement or lists an extra one (Table 27).

We evaluate State Recognition at annotated points of task progress, such as when a required value is introduced or confirmed. Active Recall uses the first prefix at which all required information is established (§4, Algorithm 1). Each prefix is scored independently, with off-task turns retained as context.

Thank you again for the constructive questions.

Thanks for the response. The clarification on the flat-ledger comparison and the details of the human annotation audit address my main questions. My rating remains weak acceptance.

Thank you for taking the time to read our response and give your assessment. We appreciate your feedback and the opportunity to clarify the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.