- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AWDM to study how far an agent should proceed before the user has completed the decision process.
Critical strengths
-- Nice and important study setup.
Critical weaknesses
-- The paper does not clearly define how the benchmark was constructed and how the data was curated. Was it adapted from existing sources? IF not, how many users were involved in the curation process? Just stating 'travel, email, and file domains' is not enough.
-- While the evaluation setup is interesting, I was not sure how different metrics are relevant and how they connect with each other in the broader picture. Rewriting the method section would be helpful.
-- Some relevant works are missing: Cocoa: Co-planning and co-execution with ai agents, Modeling Distinct Human Interaction in Web Agents
-- I believe the study setup is geniunely timely and interesting, but the paper itself can be significantly improved to show the broader implications.
Questions for authors
-- see weakness above
6 comments
W2: While the evaluation setup is interesting, I was not sure how different metrics are relevant and how they connect with each other in the broader picture. Rewriting the method section would be helpful.
R2: To address the lack of clarity regarding how the evaluation metrics relate to one another, we reorganized the Evaluation subsection in the Method section in two aspects: (i) Evaluation Framework: We establish Useful Preparation and Intent-boundary Compliance as the two core evaluation dimensions; and (ii) Metric Relationships: We clarify the hierarchy and respective roles of stage-level component metrics, stage-level joint metrics, and task-level metrics. The revisions are presented in Section 3.5, with complete metric definitions provided in Table 7 in Appendix A.3.
AWDM's evaluation centers on two interrelated requirements: whether the agent completes the preparation needed for the current user requirements, and whether it respects the user requirements, choices, and authorizations disclosed at the time of each action. These dimensions capture task progress and behavioral boundaries, respectively. For example, an agent may complete useful preparation but prematurely make a choice that the user has not yet made. Conversely, it may avoid violations by taking no action while failing to complete necessary preparation.
The evaluation framework consists of three levels:
-
Stage-level component metrics: Preparation Coverage (PC) measures the completion of current preparation targets; Safe Action Rate (SAR) measures the proportion of assessed actions that are safe; and Intent Violation Rate (IVR) measures the proportion of stages containing at least one violation. These metrics capture preparation progress, action-level safety, and stage-level violations, respectively.
-
Stage-level joint metric: Intent Success (IS) evaluates whether preparation and safety requirements are simultaneously satisfied. A stage is considered successful only when the required preparation is completed and all actions respect the current intent boundary. Authorization stages must additionally satisfy the corresponding authorization requirements.
-
Task-level metrics: Final PC and Completion measure preparation coverage and the completion of active work items at the end of a task, respectively. Task Safety measures whether any violation occurs throughout the task, Task Success aggregates joint success across stages, and Goal Attainment measures whether the final business objective is achieved.
For example, an agent may achieve the final business goal despite having violated an intent boundary earlier in the trajectory. Therefore, Goal Attainment cannot substitute for evaluating Safe Preparation throughout the execution process. Component metrics diagnose insufficient preparation and intent violations separately, the joint metric determines whether both requirements are satisfied, and task-level metrics reflect the overall execution process and final outcome.
W3: Some relevant works are missing: Cocoa: Co-planning and co-execution with ai agents, Modeling Distinct Human Interaction in Web Agents
R3: We have expanded the Related Work section in two aspects. (i) Suggested Works: We incorporate and discuss the two studies recommended by the reviewer, Cocoa: Co-Planning and Co-Execution with AI Agents and Modeling Distinct Human Interaction in Web Agents. (ii) Additional Literature Review: We further broaden our coverage of related research on evolving user intent and authorization alignment, including additional discussion of existing studies and newly added relevant work.
Specifically, Cocoa introduces interactive plans that support collaborative planning and execution between users and agents, while Modeling Distinct Human Interaction in Web Agents examines different patterns of user interaction and intervention in web-agent tasks. We also expand our discussion of UserBench and Tack et al. (2026), which study preference elicitation and evolving user intent in interactive and long-horizon tasks, respectively. In addition, we incorporate AuthorizationBench, which evaluates whether agents act within the authorization granted by users.
Dear Reviewer ocTA,
Thank you for recognizing the value of AWDM. We have carefully addressed your comments through four main improvements: (1) Benchmark Construction: We provide additional details on the public data sources, task synthesis pipeline, and human curation process. (2) Evaluation Framework: We revise the Evaluation subsection in the Method section to clarify how the metrics capture preparation progress and intent-boundary compliance. (3) Related Work: We incorporate Cocoa and Modeling Distinct Human Interaction in Web Agents to clarify AWDM's position within research on dynamic human-agent collaboration. (4) Broader Implications: We extend the experiments to four agent models and all 28 trajectory patterns, add fixed-task Completion and full-sample Goal Attainment analyses, and expand the Discussion on their implications for agent evaluation and execution strategies. These revisions have been incorporated into the revised manuscript and are detailed below. We hope they adequately address your concerns.
R1: To address the reviewer's questions about task sources and data curation, we have made three additions to the revised manuscript: (i) Data Sources: We identify the public business materials used and explain their specific purposes; (ii) Task Synthesis: We detail the process of transforming these materials into simulated tasks, multi-stage intent trajectories, and different intent-disclosure versions; and (iii) Human Curation: We clarify the number of researchers involved in task curation and the specific aspects reviewed. The relevant details are provided in the main text and Appendix A.1.
AWDM is constructed through a dedicated task synthesis pipeline based on collected and curated public business materials. It does not directly reuse task instances from existing benchmarks or real-world user data. Specifically, we refer to KAYAK and Expedia help documentation to derive travel workflows, Gmail help documentation for email operations, and Google Drive help documentation for file operations. We also draw on OSWorld for file-operation scenarios and evaluation based on application execution outcomes. These sources provide the business workflows, operation types, and rules that inform task synthesis.
Based on these materials, we first construct simulated application data, including flight inventories, email threads, attachments, and documents. We then define the business objective, initial state, and ordered intent-evolution trajectory for each task, specifying the preparation requirements, user choices, and execution authorizations at each stage. Finally, while keeping the business objective, initial state, and underlying ground-truth intent trajectory unchanged, we generate three intent-disclosure versions (L1, L2, and L3) to control the intent information available to the agent at different stages.
Task curation and review were conducted by one researcher, who examined the plausibility of business workflows, the consistency of intent evolution, and the evaluability of stage-specific preparation requirements and authorization boundaries. The agents operate on simulated data preconstructed and stored in our repository, rather than real user records. Appendix A.1 also provides a complete flight-task example illustrating the synthesis process, from business workflow specification and initial requirement setting to choice resolution, requirement revision, and execution authorization.