acceptodds
Under review as a conference paper at ICLR 2027

AWDM: Benchmarking Safe Preparation in Computer-Use Agents under Evolving User Intent

Abstract

Computer-use agents increasingly act on behalf of users in web environments. In real interactions, user intent often takes shape gradually as a task progresses, while agent actions may prematurely commit to choices that have not yet been decided. This raises a central question: how far should an agent proceed before the user has completed the decision process? A well-behaved agent should continue making progress based on confirmed information while preserving choices that remain for the user to decide. We call this capability Safe Preparation. Existing benchmarks mostly focus on one-shot intent clarification or execution under fixed constraints, and provide limited evaluation of whether agents can maintain both task progress and decision boundaries as user intent evolves. To address this gap, we introduce AWDM (Act Without Deciding for Me), a benchmark for evaluating task progress and intent-boundary compliance in multi-stage interactions. AWDM has two core designs. (a) Intent evolution organizes tasks as multi-stage intent sequences in which users progressively add, clarify, or revise their goals, with three controlled levels of intent clarity. This evaluates whether agents can adapt their actions as intent evolves and ask for clarification when critical information is missing. (b) Intent boundaries define the permissible actions at each step based on the intent and permissions disclosed so far and the current environment state, enabling the identification of assumptions or actions that exceed the user's current authorization. Extensive experiments show that final task completion does not reflect stage-wise alignment: agents complete most work items but succeed in only a minority of stages, and they are least safe in tasks that require leaving an open choice to the user. These results point to a persistent gap in preparing without deciding as user intent evolves.

Primary area:
datasets and benchmarks
Keywords:
Computer-Use Agents, Web Agents, Agent Safety, Human-AI Interaction, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ocTAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AWDM to study how far an agent should proceed before the user has completed the decision process.

Critical strengths

-- Nice and important study setup.

Critical weaknesses

-- The paper does not clearly define how the benchmark was constructed and how the data was curated. Was it adapted from existing sources? IF not, how many users were involved in the curation process? Just stating 'travel, email, and file domains' is not enough.

-- While the evaluation setup is interesting, I was not sure how different metrics are relevant and how they connect with each other in the broader picture. Rewriting the method section would be helpful.

-- Some relevant works are missing: Cocoa: Co-planning and co-execution with ai agents, Modeling Distinct Human Interaction in Web Agents

-- I believe the study setup is geniunely timely and interesting, but the paper itself can be significantly improved to show the broader implications.

Questions for authors

-- see weakness above

6 comments

Dear Reviewer ocTA,

Thank you for recognizing the value of AWDM. We have carefully addressed your comments through four main improvements: (1) Benchmark Construction: We provide additional details on the public data sources, task synthesis pipeline, and human curation process. (2) Evaluation Framework: We revise the Evaluation subsection in the Method section to clarify how the metrics capture preparation progress and intent-boundary compliance. (3) Related Work: We incorporate Cocoa and Modeling Distinct Human Interaction in Web Agents to clarify AWDM's position within research on dynamic human-agent collaboration. (4) Broader Implications: We extend the experiments to four agent models and all 28 trajectory patterns, add fixed-task Completion and full-sample Goal Attainment analyses, and expand the Discussion on their implications for agent evaluation and execution strategies. These revisions have been incorporated into the revised manuscript and are detailed below. We hope they adequately address your concerns.

W1: The paper does not clearly define how the benchmark was constructed and how the data was curated. Was it adapted from existing sources? IF not, how many users were involved in the curation process? Just stating 'travel, email, and file domains' is not enough.

R1: To address the reviewer's questions about task sources and data curation, we have made three additions to the revised manuscript: (i) Data Sources: We identify the public business materials used and explain their specific purposes; (ii) Task Synthesis: We detail the process of transforming these materials into simulated tasks, multi-stage intent trajectories, and different intent-disclosure versions; and (iii) Human Curation: We clarify the number of researchers involved in task curation and the specific aspects reviewed. The relevant details are provided in the main text and Appendix A.1.

AWDM is constructed through a dedicated task synthesis pipeline based on collected and curated public business materials. It does not directly reuse task instances from existing benchmarks or real-world user data. Specifically, we refer to KAYAK and Expedia help documentation to derive travel workflows, Gmail help documentation for email operations, and Google Drive help documentation for file operations. We also draw on OSWorld for file-operation scenarios and evaluation based on application execution outcomes. These sources provide the business workflows, operation types, and rules that inform task synthesis.

Based on these materials, we first construct simulated application data, including flight inventories, email threads, attachments, and documents. We then define the business objective, initial state, and ordered intent-evolution trajectory for each task, specifying the preparation requirements, user choices, and execution authorizations at each stage. Finally, while keeping the business objective, initial state, and underlying ground-truth intent trajectory unchanged, we generate three intent-disclosure versions (L1, L2, and L3) to control the intent information available to the agent at different stages.

Task curation and review were conducted by one researcher, who examined the plausibility of business workflows, the consistency of intent evolution, and the evaluability of stage-specific preparation requirements and authorization boundaries. The agents operate on simulated data preconstructed and stored in our repository, rather than real user records. Appendix A.1 also provides a complete flight-task example illustrating the synthesis process, from business workflow specification and initial requirement setting to choice resolution, requirement revision, and execution authorization.

W2: While the evaluation setup is interesting, I was not sure how different metrics are relevant and how they connect with each other in the broader picture. Rewriting the method section would be helpful.

R2: To address the lack of clarity regarding how the evaluation metrics relate to one another, we reorganized the Evaluation subsection in the Method section in two aspects: (i) Evaluation Framework: We establish Useful Preparation and Intent-boundary Compliance as the two core evaluation dimensions; and (ii) Metric Relationships: We clarify the hierarchy and respective roles of stage-level component metrics, stage-level joint metrics, and task-level metrics. The revisions are presented in Section 3.5, with complete metric definitions provided in Table 7 in Appendix A.3.

AWDM's evaluation centers on two interrelated requirements: whether the agent completes the preparation needed for the current user requirements, and whether it respects the user requirements, choices, and authorizations disclosed at the time of each action. These dimensions capture task progress and behavioral boundaries, respectively. For example, an agent may complete useful preparation but prematurely make a choice that the user has not yet made. Conversely, it may avoid violations by taking no action while failing to complete necessary preparation.

The evaluation framework consists of three levels:

  • Stage-level component metrics: Preparation Coverage (PC) measures the completion of current preparation targets; Safe Action Rate (SAR) measures the proportion of assessed actions that are safe; and Intent Violation Rate (IVR) measures the proportion of stages containing at least one violation. These metrics capture preparation progress, action-level safety, and stage-level violations, respectively.

  • Stage-level joint metric: Intent Success (IS) evaluates whether preparation and safety requirements are simultaneously satisfied. A stage is considered successful only when the required preparation is completed and all actions respect the current intent boundary. Authorization stages must additionally satisfy the corresponding authorization requirements.

  • Task-level metrics: Final PC and Completion measure preparation coverage and the completion of active work items at the end of a task, respectively. Task Safety measures whether any violation occurs throughout the task, Task Success aggregates joint success across stages, and Goal Attainment measures whether the final business objective is achieved.

For example, an agent may achieve the final business goal despite having violated an intent boundary earlier in the trajectory. Therefore, Goal Attainment cannot substitute for evaluating Safe Preparation throughout the execution process. Component metrics diagnose insufficient preparation and intent violations separately, the joint metric determines whether both requirements are satisfied, and task-level metrics reflect the overall execution process and final outcome.

W3: Some relevant works are missing: Cocoa: Co-planning and co-execution with ai agents, Modeling Distinct Human Interaction in Web Agents

R3: We have expanded the Related Work section in two aspects. (i) Suggested Works: We incorporate and discuss the two studies recommended by the reviewer, Cocoa: Co-Planning and Co-Execution with AI Agents and Modeling Distinct Human Interaction in Web Agents. (ii) Additional Literature Review: We further broaden our coverage of related research on evolving user intent and authorization alignment, including additional discussion of existing studies and newly added relevant work.

Specifically, Cocoa introduces interactive plans that support collaborative planning and execution between users and agents, while Modeling Distinct Human Interaction in Web Agents examines different patterns of user interaction and intervention in web-agent tasks. We also expand our discussion of UserBench and Tack et al. (2026), which study preference elicitation and evolving user intent in interactive and long-horizon tasks, respectively. In addition, we incorporate AuthorizationBench, which evaluates whether agents act within the authorization granted by users.

Reviewer_dncCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AWDM (Act Without Deciding for Me), a benchmark evaluating whether computer-use agents can make progress while respecting evolving user intent and authorization boundaries. It evaluates agents across 336 tasks in travel, email, and file-management domains, covering four types of intent evolution: Specification, Resolution, Revision, and Authorization.

Experiments with four language models reveal a substantial gap between final task completion (51.0–80.8%) and stage-wise Task Success (20.6–33.7%). The results highlight that current agents frequently make premature decisions, particularly when user choices remain unresolved, demonstrating the importance of evaluating safety throughout an interaction rather than solely through final outcomes.

Critical strengths

Relevant and well-motivated problem. The paper addresses an important limitation of existing agent benchmarks: evaluating whether agents can make useful progress without prematurely committing to decisions that users have not authorized. The distinction between task completion and safe preparation is meaningful, particularly for real-world computer-use agents.

Well-structured evaluation framework. The benchmark systematically models different types of intent evolution and levels of intent clarity. The stage-wise evaluation provides a more fine-grained assessment than final-outcome metrics, revealing failure modes that would otherwise go unnoticed.

Critical weaknesses
  1. Limited realism of intent evolution. User intent changes follow predefined trajectories and fixed checkpoints rather than emerging naturally from interactions. Additionally, the simulated user cannot form new preferences or decisions in response to agent behavior. This limits how well the benchmark captures realistic, adaptive human-agent interactions.

  2. Concerns about evaluation reliability and metric design. The benchmark relies on predefined action boundaries and automated judgments, with only limited human validation (88.3% agreement from a single reviewer). Additionally, the strict stage-wise success metric penalizes any violation or incomplete preparation, potentially conflating different failure modes. More extensive validation and analysis would strengthen confidence in the reported results.

Questions for authors

See weaknesses

2 comments

Dear Reviewer dncC,

Thank you for your review and for recognizing the value of our proposed AWDM. We have carefully addressed your suggestions through two main improvements: (1) Intent Evolution Protocol: We clarify the rationale for predefined intent trajectories and unified stage-transition rules, as well as the scope of the current interaction setting. (2) Evaluation Reliability and Metric Design: We introduce a second independent reviewer, provide additional details on the human audit, and clarify how the component and joint metrics work together to evaluate Safe Preparation. These revisions have been incorporated into the paper and are summarized in the responses below. We hope they adequately address your concerns.

W1: Limited realism of intent evolution. User intent changes follow predefined trajectories and fixed checkpoints rather than emerging naturally from interactions. Additionally, the simulated user cannot form new preferences or decisions in response to agent behavior. This limits how well the benchmark captures realistic, adaptive human-agent interactions.

R1: Our predefined intent-update protocol serves two purposes: (i) to study user intent changes independently of agent task progress, and (ii) to ensure that different models encounter the same intent-update sequence for controlled comparison.

In real-world interactions, users may update their intent in response to new external information, changing needs, or reconsideration of existing choices while the agent is still working. We therefore treat intent updates as external events rather than triggering them based on task completion. Otherwise, differences in agent execution efficiency would determine when new requirements are disclosed, causing different models to encounter different intent trajectories and preparation opportunities.

Specifically, we use an interaction-step-based intent clock, triggering updates at steps 5, 10, 15, and 20. If the agent explicitly ends the current stage early, the protocol advances directly to the next pending checkpoint. This gives all models the same interaction-step budget for each stage, while preventing differences in inference time or tool latency from affecting the update schedule. The five-step interval is an operational setting for controlled and reproducible evaluation, rather than an assumption that real users update their intent at fixed intervals.

Regarding user adaptivity, the current protocol allows simulated users to progressively disclose existing information according to predefined trajectories and answer the agent's questions. However, users do not dynamically form new preferences or make new decisions in response to the agent's intermediate behavior. Accordingly, AWDM evaluates how agents prepare for and respond to intent changes that occur independently of their execution progress. Different agents may still complete different amounts of preparation within the same number of steps; thus, the evaluation measures overall performance under a unified intent-update schedule rather than isolating adaptation ability at an identical preparation state. We clarify this scope and its implications in the revised Method and Discussion sections.

W2: Concerns about evaluation reliability and metric design. The benchmark relies on predefined action boundaries and automated judgments, with only limited human validation (88.3% agreement from a single reviewer). Additionally, the strict stage-wise success metric penalizes any violation or incomplete preparation, potentially conflating different failure modes. More extensive validation and analysis would strengthen confidence in the reported results.

R2: We addressed these concerns through two revisions. (i) Human Validation: We added a second independent reviewer and report task-level and violation-type agreement, as well as the impact on Task Safety. (ii) Metric Design: We clarify how component metrics distinguish preparation failures from intent violations and how Intent Success jointly evaluates both. These revisions are reflected in Appendix C.6 and Section 3.5, respectively.

(i) Independent human review and agreement analysis.

We randomly sampled 60 complete execution records, with 20 from each of the travel, email, and file domains. Two human reviewers independently classified each record as safe or unsafe based on the intent disclosed at each action, the applicable authorization, and the action outcome. Task identities and the evaluator's original judgments were retrieved from the saved execution records.

The evaluator agreed with Reviewer 1 on 53/60 tasks (88.33%) and Reviewer 2 on 52/60 tasks (86.67%), with Cohen's κ values of 0.7482 and 0.7143, respectively.

ComparisonAgreementCohen's κFalse PositivesFalse Negatives
Evaluator vs. Reviewer 153/60 (88.33%)0.748261
Evaluator vs. Reviewer 252/60 (86.67%)0.714362

The two reviewers agreed on 47/60 tasks (78.33%), with Cohen's κ = 0.5063. Their cross-classification is shown below.

Reviewer 1 / Reviewer 2SafeUnsafe
Safe347
Unsafe613

We further analyzed agreement across individual violation categories (V1, V2, and V3). Each category was evaluated over all 60 tasks, and a task could have multiple violation types. Evaluator-assigned violation types were retained when the corresponding reviewer judged a task unsafe; otherwise, they were counted as false positives. False negatives identified during human review were assigned to V1.

Violation TypeReviewer 1 AgreementFalse PositivesFalse NegativesReviewer 2 AgreementFalse PositivesFalse Negatives
V156/60 (93.33%)3154/60 (90.00%)42
V259/60 (98.33%)1059/60 (98.33%)10
V358/60 (96.67%)2057/60 (95.00%)30

V2 achieves the highest agreement with both reviewers.

We also quantified the impact of evaluator disagreements on Task Safety. The evaluator classifies 36/60 tasks as safe, compared with 41/60 and 40/60 according to Reviewers 1 and 2, respectively. The corresponding Task Safety estimates are 60.00%, 68.33%, and 66.67%. Thus, the evaluator's estimates are 8.33 and 6.67 percentage points lower than those of the two reviewers. This difference reflects the predominance of false positives (6 against each reviewer) over false negatives (1 and 2, respectively), indicating that the evaluator tends to classify more tasks as unsafe than the human reviewers.

(ii) Clarification of metric relationships and stage-wise success.

We reorganized Section 3.5 around two complementary dimensions of Safe Preparation: Useful Preparation and Intent-boundary Compliance.

Preparation Coverage (PC) measures the completion of preparation requirements, Safe Action Rate (SAR) measures action-level safety, and Intent Violation Rate (IVR) captures the proportion of stages containing at least one violation. These component metrics separately diagnose insufficient preparation and intent violations.

Intent Success (IS) provides a joint evaluation. A stage is successful only when its preparation requirements are completed and all actions respect the current intent boundary. Authorization stages must additionally satisfy the corresponding authorization requirements. Thus, a stage may fail IS due to insufficient preparation, an intent violation, or both, while the component metrics help identify the source of failure.

For example, in Gemini's results, the violation rate in Authorization stages is lower than in Revision stages (2.8% vs. 6.5%), while the corresponding joint success rate is also lower (16.0% vs. 39.6%). This demonstrates that fewer violations do not necessarily indicate better stage-level performance, as preparation requirements may remain incomplete.

At the task level, Task Safety, Task Success, and Goal Attainment further distinguish overall safety, joint execution success, and final business-goal achievement. We have clarified these metric definitions and their relationships in the revised Method section.

Reviewer_sKwLOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

AWDM tests whether computer-use agents can make progress while leaving undecided choices and unauthorized actions to the user. Its 336 tasks change requirements, choices, or permissions during an interaction. Tests of four models show that completing work often hides earlier unsafe actions or incomplete preparation.

Critical strengths
  • The benchmark separates useful preparation from permission to commit an action. This captures failures that a final-result check can miss. (Sections 3.2 and 3.5.)
  • The same underlying tasks are tested with different amounts of disclosed information, allowing a controlled study of intent clarity. (Section 3.3; Appendix C.2.)
  • Intermediate drafts, documents, and candidate lists provide observable evidence of progress. The flight-booking examples clearly illustrate why earlier violations still matter after a correct final result. (Appendix A.1; Section 4.2.)
Critical weaknesses
  1. Intent updates follow an artificial schedule. Updates arrive every five agent steps, regardless of how much work the agent has completed. The paper acknowledges this design, but it can give slower agents less time to prepare and limits conclusions about natural user interaction.
  2. The human check of the evaluator is limited. Appendix C.6 reports 88.3% agreement with one human reviewer, but does not state the sample size, sampling procedure, or agreement by violation type. This makes it difficult to assess how evaluation errors affect the reported safety scores.
  3. The intent-clarity finding has limited coverage. It is tested with Qwen 3.8 Flash on 60 tasks covering ten of the 28 trajectory patterns. The matched design is useful, but it does not establish whether the same effects hold for the other three models or the remaining patterns.
  4. Completion comparisons use substantially different eligible samples. The L1, L2, and L3 Completion scores use 36, 56, and 59 episodes, respectively. Reporting the same tasks at every level would help separate the effect of clearer intent from changes in which episodes enter this metric.
Questions for authors

NA

6 comments

Dear Reviewer sKwL,

Thank you for your review and for recognizing the value of AWDM. We have carefully addressed your comments through four main revisions: (1) Intent Evolution Protocol: We clarify the rationale for decoupling intent updates from agent progress and the scope of the fixed-step schedule. (2) Human Validation: We introduce a second independent reviewer and report inter-rater agreement and additional audit details. (3) Intent-Clarity Evaluation: We extend the experiments to four models and all 28 trajectory patterns. (4) Completion Comparability: We conduct a fixed-task comparison across L1, L2, and L3 and an additional Goal Attainment analysis over all 60 paired tasks. These revisions have been incorporated into the paper and are summarized in the responses below. We hope they adequately address your concerns.

W1: Intent updates follow an artificial schedule. Updates arrive every five agent steps, regardless of how much work the agent has completed. The paper acknowledges this design, but it can give slower agents less time to prepare and limits conclusions about natural user interaction.

R1: Our predefined intent-update schedule serves two purposes: (i) to study intent evolution independently of agent execution progress, and (ii) to ensure that different models encounter the same sequence of intent updates for controlled comparison.

Our intent-update protocol is designed to capture user intent changes that occur independently of the agent's execution progress and to ensure consistent comparisons across models. In real interactions, users may revise their requirements while an agent is still working, for example, after receiving new information or reconsidering an earlier choice. If intent updates were triggered by task completion, their timing would depend on each agent's execution efficiency. Different models would then receive new requirements at different task states and preparation levels, making their results less directly comparable.

We therefore use a unified interaction-step-based schedule, with intent updates triggered every five agent steps. If an agent explicitly ends the current stage earlier, the protocol advances to the next pending checkpoint. This ensures that all models encounter the same intent-update sequence and receive the same interaction-step budget at each stage, regardless of differences in inference time or tool latency. The five-step interval is an operational choice for controlled and reproducible evaluation, rather than an assumption about how frequently users change their intent in practice.

This design also defines the scope of our evaluation. Since different agents may complete different amounts of preparation within the same step budget, the results reflect their overall performance in preparation, task execution, and adaptation under a shared intent-update schedule. They do not isolate adaptation ability under identical preparation states. We have clarified this distinction and the scope of the protocol in the revised Method and Discussion sections.

W2: The human check of the evaluator is limited. Appendix C.6 reports 88.3% agreement with one human reviewer, but does not state the sample size, sampling procedure, or agreement by violation type. This makes it difficult to assess how evaluation errors affect the reported safety scores.

R2: We strengthened the human validation in three aspects: (i) Independent Human Review: We introduced a second reviewer to independently assess the same 60 execution records; (ii) Agreement Analysis: We report the sampling procedure, agreement rates, Cohen's κ, and disagreement counts at both task and violation-type levels; and (iii) Impact on Safety Estimates: We quantify how evaluator disagreements affect the reported Task Safety results. The expanded analysis is provided in Appendix C.6.

(1) Sampling and independent human review.

We randomly sampled 60 execution records, with 20 records from each of the travel, email, and file domains. Two human reviewers independently assessed every record and classified it as safe or unsafe. Task identities and the evaluator's original judgments were retrieved from the saved execution records. The two reviewers' judgments served as independent human references for evaluating the automated evaluator.

(2) Task-level agreement.

The automated evaluator agreed with human reviewer 1 on 53 of 60 tasks (88.33%) and with human reviewer 2 on 52 of 60 tasks (86.67%). We also report Cohen's κ and the direction of disagreements. A false positive (FP) occurs when the evaluator labels a task as unsafe while the human reviewer labels it as safe; a false negative (FN) occurs when the evaluator labels a task as safe while the human reviewer labels it as unsafe.

ComparisonAgreementCohen's κFalse PositivesFalse Negatives
Evaluator vs. Human Reviewer 153/60 (88.33%)0.748261
Evaluator vs. Human Reviewer 252/60 (86.67%)0.714362

The two human reviewers agreed on 47 of 60 tasks (78.33%), with Cohen's κ = 0.5063. Their cross-classification is shown below.

Human Reviewer 1 / Reviewer 2SafeUnsafe
Safe347
Unsafe613

(3) Agreement by violation type.

We further examined evaluator agreement for individual violation categories (V1, V2, and V3). Each category is evaluated over all 60 tasks, and multiple violation types may be assigned to a single task. Evaluator-assigned violation types are retained when the corresponding human reviewer judges the task unsafe; if the reviewer judges the task safe, the corresponding category is counted as a false positive. False negatives identified during human review are assigned to V1.

Violation TypeReviewer 1 AgreementFalse PositivesFalse NegativesReviewer 2 AgreementFalse PositivesFalse Negatives
V156/60 (93.33%)3154/60 (90.00%)42
V259/60 (98.33%)1059/60 (98.33%)10
V358/60 (96.67%)2057/60 (95.00%)30

The results show different agreement levels across violation categories, with V2 achieving the highest agreement with both reviewers. The false-positive and false-negative counts further identify the categories in which automated judgments differ from human assessments.

(4) Impact on Task Safety estimates.

The evaluator classifies 36 of the 60 tasks as safe, compared with 41 according to reviewer 1 and 40 according to reviewer 2. Accordingly, the Task Safety estimates are 60.00%, 68.33%, and 66.67%, respectively. The evaluator's estimate is therefore 8.33 and 6.67 percentage points lower than those of the two human reviewers. This difference is consistent with the task-level disagreement counts. The evaluator produces six false positives against each reviewer, compared with one and two false negatives, respectively. Its disagreements therefore predominantly involve classifying tasks as unsafe when human reviewers judge them safe, resulting in lower estimated Task Safety.

Overall, the additional audit provides task-level and violation-type agreement measurements, while quantifying the direction and magnitude of evaluator disagreements on aggregate Task Safety. These results provide a more detailed assessment of the evaluator's consistency with independent human judgments and the potential impact of evaluation errors on the reported safety results.

W3: The intent-clarity finding has limited coverage. It is tested with Qwen 3.8 Flash on 60 tasks covering ten of the 28 trajectory patterns. The matched design is useful, but it does not establish whether the same effects hold for the other three models or the remaining patterns.

R3: We expanded the intent-clarity evaluation from one model and ten trajectory patterns to four models and all 28 trajectory patterns, yielding 1,008 valid episode results.

Specifically, we evaluated Qwen 3.8 Flash, DeepSeek V4.1 Flash, Gemini 3.8 Flash, and GPT-6 Luna. Each model was evaluated on 84 business tasks under each of L1, L2, and L3, covering all 28 intent-evolution patterns.

(a) Consistency across models.

The expanded results show consistent improvements in safety and task performance as intent disclosure increases. The table reports all main metrics under the three clarity conditions. All values are percentages.

ModelIntent ClarityPreparation CoverageSafe Action RateIntent Violation Rate ↓Intent SuccessFinal PCTask SafetyCompletionTask SuccessGoal Attainment
Qwen 3.8 FlashL126.0071.9034.432.7828.7929.7639.023.1711.90
L243.3777.6736.7218.0650.1648.8144.8417.4629.76
L338.9187.5620.8124.0743.7758.3364.3524.9051.19
DeepSeek V4.1 FlashL130.5076.6440.462.3136.9033.3343.253.5713.10
L242.2687.2031.6414.8153.5346.4349.0815.7726.19
L344.2092.4617.3425.9349.8069.0564.8628.2753.57
Gemini 3.8 FlashL132.9276.2935.186.0238.8128.5738.226.7514.29
L242.5085.5437.5014.8147.5833.3344.3714.7826.19
L343.7194.5512.6231.0249.4669.0578.9831.8567.86
GPT-6 LunaL131.0680.1232.753.7036.8335.7145.655.2623.81
L244.6680.1534.5415.7450.9940.4846.3215.0834.52
L338.1193.4418.0320.8342.4467.8657.0321.6347.62

Three consistent trends emerge across the four models:

  • Task Safety improves from L1 to L3 by 28.57, 35.72, 40.48, and 32.15 percentage points for Qwen, DeepSeek, Gemini, and GPT, respectively.
  • Intent Violation Rate decreases from L2 to L3 by 15.91, 14.30, 24.88, and 16.51 percentage points, respectively.
  • Task Success increases from L1 to L3 by 21.73, 24.70, 25.10, and 16.37 percentage points, respectively. Completion and Goal Attainment also increase across all four models.

(b) Coverage across trajectory patterns.

All 28 intent-evolution patterns are included in the expanded evaluation, covering Specification, Resolution, Revision, Authorization, and their combinations. Each model is evaluated on 84 tasks per clarity level. This extends the original finding from ten selected patterns to the complete trajectory set.

Overall, the expanded evaluation shows that the observed relationship between intent clarity, safety, and task performance holds across four models and all 28 trajectory patterns, supporting the generality of the original finding.

Comments on the paper

2 comments

Dear Reviewers and Area Chairs,

We sincerely thank the reviewers and area chairs for their time and thoughtful feedback on our submission. We appreciate the recognition of AWDM's value as a benchmark for evaluating how agents prepare for and respond to progressively evolving user intents while respecting users' disclosed choices and authorization boundaries. We have addressed the comments point by point in our individual responses and incorporated the corresponding revisions into the manuscript. Below, we summarize the key clarifications, additional experiments, and analyses included in our rebuttal.

Expanded Intent-Clarity Evaluation Across Four Models and All Trajectory Patterns

Following the suggestions of Reviewers sKwL and ocTA, we have substantially expanded the intent-clarity evaluation:

  • We extend the evaluation from one model and ten trajectory patterns to four models—Qwen 3.8 Flash, DeepSeek V4.1 Flash, Gemini 3.8 Flash, and GPT-6 Luna—covering all 28 intent-evolution patterns. Each model is evaluated on 84 tasks per clarity level, yielding 1,008 valid episodes across four models and three clarity levels.
  • Across the four models, Task Safety increases by 28.57–40.48 percentage points from L1 to L3, while Task Success increases by 16.37–25.10 percentage points. These consistent improvements across all 28 patterns demonstrate the generality of the intent-clarity findings.

Strengthened Human Validation and Clarified the Evaluation Framework

In response to Reviewers sKwL, dncC, and ocTA, we have strengthened the human validation and clarified the relationships among evaluation metrics:

  • We introduce a second independent reviewer to assess the same 60 sampled trajectories, with 20 trajectories from each of the travel, email, and file domains. The automated evaluator agrees with the first and second reviewers on 53/60 (88.3%) and 52/60 (86.7%) trajectories, respectively. The Cohen's κ between the two reviewers is 0.506. The sampling procedure, annotation criteria, and disagreement analysis are provided in Appendix C.6.
  • We reorganize the evaluation framework around two core dimensions—Useful Preparation and Intent-boundary Compliance—and clarify the hierarchy of stage-level component metrics (PC, SAR, IVR), the stage-level joint metric (IS), and task-level metrics. The revised framework is presented in Section 3.5, with complete metric definitions in Table 8 of Appendix A.3.

Task Completion and Goal-Attainment Analysis

Following Reviewer sKwL's concern about differences in eligible samples across intent-clarity conditions, we conduct two additional analyses:

  • On the same 35 tasks for which Completion is applicable under all three clarity conditions, Completion increases from 30.00% (L1) to 35.24% (L2) and 52.29% (L3), controlling for task composition.
  • Across all 60 paired tasks, Goal Attainment increases from 13.33% (L1) to 25.00% (L2) and 46.67% (L3). Both analyses show consistent improvements in active-work-item completion and final goal attainment with fuller intent disclosure.

Clarified Intent-Evolution Protocol, Benchmark Construction, and Related Work

Following the suggestions of Reviewers sKwL, dncC, and ocTA, we have clarified the experimental protocol and expanded the descriptions of benchmark construction and related work:

  • We clarify the rationale for the fixed interaction-step intent clock, with updates at steps 5, 10, 15, and 20. This design decouples intent updates from task-stage completion and provides the same update checkpoints across models. We also explicitly acknowledge that the current simulated user does not dynamically form new preferences or decisions in response to intermediate agent behavior.
  • We detail the public business materials used, including KAYAK and Expedia help documentation for travel, Gmail help documentation for email, Google Drive help documentation for file operations, and OSWorld for file-operation scenarios and application-based evaluation. We further describe the task synthesis pipeline and clarify that task curation and review were conducted by one researcher. These details are provided in the main text and Appendix A.1.
  • We incorporate Cocoa and Modeling Distinct Human Interaction in Web Agents into the Related Work and Introduction, clarifying AWDM's relationship to existing research on dynamic human-agent collaboration. We also expand the Discussion to explain the implications of multi-stage intent evolution for agent evaluation and interaction strategies.

Dear Area Chairs,

We sincerely thank all reviewers and area chairs for their thoughtful feedback and guidance throughout the discussion. We are encouraged that the reviewers recognized the importance of AWDM's research problem and the value of its evaluation framework, particularly its distinction between making useful task progress and respecting users' unresolved decisions and authorization boundaries. Reviewer dncC highlighted the relevance of this problem and the value of evaluating agents throughout the interaction, while Reviewer ocTA also acknowledged that the study setup is timely and interesting. In response to the reviewers' concerns, we have carefully revised the manuscript and conducted additional experiments and analyses. We summarize the main improvements below.

(1) Benchmark Construction and Intent Evolution Protocol. We have substantially expanded the description of AWDM's task construction and curation process. Specifically, we identify the public sources used to derive business workflows, including KAYAK and Expedia documentation for travel, Gmail documentation for email, and Google Drive documentation for file operations, with OSWorld informing file-operation scenarios and execution-based evaluation. We clarify how these materials are transformed into simulated application data, business objectives, and multi-stage intent trajectories, and how one researcher reviewed task plausibility, intent consistency, and stage-specific preparation and authorization requirements. We also clarify the rationale and limitations of the predefined five-step intent-update schedule: it enables controlled comparisons under a shared sequence of intent changes, while the current benchmark does not model users dynamically forming new preferences in response to agent behavior.

(2) Evaluation Reliability and Metric Framework. To strengthen the human validation of our evaluator, we introduced a second independent reviewer and audited 60 execution records, with 20 samples from each of the travel, email, and file domains. We report task-level agreement, Cohen's κ, disagreement counts, and agreement across individual violation types. We also quantify the impact of evaluator disagreements on Task Safety, showing that the automated evaluator produces more conservative safety estimates than either human reviewer. In addition, we reorganized the evaluation framework around two dimensions: Useful Preparation and Intent-boundary Compliance. We clarify how component metrics (PC, SAR, and IVR), the joint stage-level metric (IS), and task-level metrics capture different aspects of agent behavior, rather than treating final task completion as a sufficient measure of safe execution.

(3) Expanded Intent-Clarity Evaluation. Following the concern about limited model and trajectory coverage, we extended the intent-clarity experiments from one model and ten trajectory patterns to four models and all 28 patterns, yielding 1,008 valid episode results. The expanded results show consistent improvements in Task Safety and Task Success as more intent information is disclosed across all four models. We further conducted a fixed-task Completion comparison using the same 35 tasks across L1, L2, and L3, with Completion increasing from 30.00% to 52.29%. We also report Goal Attainment across all 60 paired tasks, increasing from 13.33% to 46.67%. These additional analyses help distinguish improvements in preparation and task outcomes from differences in metric eligibility.

(4) Broader Implications and Related Work. We incorporated the suggested works on collaborative planning and human-agent interaction, including Cocoa and Modeling Distinct Human Interaction in Web Agents, and expanded our discussion of related research on evolving user intent and authorization. We also strengthened the Discussion to explain why agent evaluation should consider preparation progress and intent-boundary compliance alongside final outcomes. In particular, AWDM highlights the distinction between undisclosed information, which agents can seek to clarify, and unresolved user decisions or authorization, which agents should leave to users while continuing permissible preparation.

We have incorporated these additional experiments, clarifications, and discussions into the revised manuscript. We believe these revisions provide a clearer account of AWDM's construction, a more thoroughly examined evaluation framework, and broader evidence for the relationship between intent clarity, safe preparation, and task performance. We sincerely hope that the additional evidence and clarifications will be taken into consideration in the final assessment.

Thank you for your time and consideration.

Best regards,

Submission #40996 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.