- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates interactive task alignment and propose an evaluation framework. The framework requires models to interact with the user and then evaluates alignment with the user's true intention by candidate ranking.
Critical strengths
- The problem is framed well, and the cross-domain evaluation provides a useful perspective beyond conventional task-completion benchmarks.
- The human-transcript replay experiment is informative.
Critical weaknesses
- The assistant cannot access candidate tasks during interaction but only receives their complete specifications during evaluation. Therefore, the metric combines information elicitation with identification of emerging alternatives. Ranking the correct task first does not necessarily mean that the assistant had formed the correct actionable understanding when it committed.
- The evaluation conflates clarification behavior with discovering the relevant task space. If the assistant does not observe the alternative options or explore the environment, it may not be clear which differences need to be clarified. Some perturbed attributes may also represent implementation choices that could reasonably be resolved through defaults or environment inspection. Therefore, low recovery scores and stable stopping tendencies do not uniquely demonstrate deficient interaction policies. Controlled experiments with fixed interaction budgets and access to task dimensions or environment exploration would help distinguish these explanations.
Questions for authors
Please see Critical Weaknesses above.
5 comments
Response to authors
I appreciate the quick response and the additional validation and turn-budget analyses.
The reported agreement rates show how often the final task specification and execution trace match the top-ranked candidate across the audited episodes. Could you also report, separately for GDPVal and Terminal-Bench, how often the final specification and execution trace each match the ground-truth task in episodes where that task was ranked first? This would clarify how reliably a correct ranking reflects what the assistant committed to and attempted to execute.
Your point that the evaluation captures both clarification and task discovery also made me wonder how Section 4.4 draws the line between communication and inference. Suppose a user asks for headphones for running. An assistant that knows about noise-isolating earbuds but is unaware of open-ear headphones might never ask whether the user wants to block out distractions or stay aware of their surroundings. If a gap in world knowledge keeps the assistant from recognizing what to ask, how would you account for that when interpreting the human advantage in Section 4.4?
Thank you for the follow-up questions. We address both below.
1: Agreement with ground-truth task when ranked first
As asked, we report below how often the final specification and execution trace match the rank-1 task on a random subset of n=40 episodes where the ground-truth task was correctly ranked first.
| Analysis | Domain | Agreement with ground-truth task at rank-1 |
|---|---|---|
| (1) Task spec the assistant proposes at end | Terminal-Bench | 0.98 ± 0.01 |
| (1) Task spec the assistant proposes at end | GDPVal | 0.91 ± 0.03 |
| (2) Execution trace | Terminal-Bench | 0.95 ± 0.03 |
| (2) Execution trace | GDPVal | 0.89 ± 0.03 |
Same as with any rank-1 task, we see that when the correct task is ranked first, the assistant also largely commits to and executes the correct task. We will shortly add this to the validity paragraph in lines 229-35, and the Appendix.
2: World knowledge v.s. interaction in Section 4.4
We agree that knowledge in general affects interaction ability, where as in your example knowing about which attributes exist and how they can vary can affect what questions the assistant asks. To account for knowledge, we provide two analyses.
(1) Knowledge control. First we provide a knowledge control prior to any interaction, where for each of the n=15 products in the human assistant study, we give the {human, model} assistant the user’s initial intent, along with the product category, and ask the assistant to list out the attributes relevant to the user’s task. We compare this list against a ground-truth attribute list that we curate manually from the full candidate set with LLM assistance.
(2) Attributes asked in conversation. We also analyze the task alignment transcripts to identify and measure the proportion of ground-truth attributes the assistant asks the user about in the conversation.
We use Deepseek-V-4.1 as Judge for evaluating against the ground-truth attributes in both analyses (1), (2). We test on all 14 models and a set of n=45 new humans different from the original human assistant study. Results are reported below.
| Assistant | Knowledge control | Attributes asked in conversation | Attributes asked, forced ≥8 turns |
|---|---|---|---|
| Human assistants | 0.23 ± 0.06 | 0.53 ± 0.03 | — |
| Model assistants | 0.64 ± 0.03 | 0.26 ± 0.02 | 0.45 ± 0.03 |
On the knowledge control, we first see that models recall more relevant task attributes than humans, which is consistent with prior work showing that LLMs exceed non-expert human baselines on general knowledge benchmarks [1][2]. Despite this, humans still cover more task dimensions than models in the conversation (0.53 vs. 0.26 base). Models ask about only 35% of the task dimensions they themselves initially listed, further reinforcing interaction rather than knowledge as the gap.
We note that the human participants in the knowledge control are different from the participants in the human assistant study (i.e. population-level comparison), and that factors such as small sample size and lack of human effort may have resulted in biased underestimates of human knowledge. However, our communication result is nonetheless supported by the observation that models know attributes but do not ask about them. We have updated the interaction framing in Section 4.4 with the role of world knowledge, and added both analyses, with full details in the Appendix.
Thank you again for your thoughtful review. We believe these clarifications and additions have made the paper stronger, and we hope they address your concerns.
[1] Hendrycks et al., Measuring Massive Multitask Language Understanding, ICLR 2021
[2] Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark, COLM 2024.
Thank you for the constructive review. We address your concerns below.
1: Validation that assistant actually commits to the recovered task
As in pJFM, we provide support that the recovered task in our setting is the task that the assistant actually commits to and executes. We provide two analyses: (1) first we audit the task spec proposed at the end of the task alignment episode against the rank-1 task, and additionally (2) we have the model execute the task based on the task alignment episode and we measure whether the execution trace aligns with the rank-1 task. We exclude Shopping from these two validations, since Shopping already maps to execution (i.e. selecting an option). In n=140 episodes from GDPVal and Terminal-Bench across n=20 tasks and 3 models selected randomly, we find agreement ranges from 0.75-0.96 with the recovered rank-1 task in both the (1) proposed task spec and (2) the execution trace. We use Deepseek-V4.1 as Judge for measuring agreement.
Furthermore, to validate our judge, we use our judge to evaluate the task spec and execution traces against each of the 14 non-rank-1 tasks, and we find low average agreement scores over all non-rank-1 tasks.
The two validations for task spec proposal and execution trace are added to the paragraph containing the other task validity checks in lines 229-35, with full details in the Appendix.
2: Clarification behavior and task discovery
We agree that our evaluation jointly measures both clarification behavior and task discovery. We would like to clarify that we follow the established definitions of task alignment in [1][2] where task alignment is defined as resolving the intended task from the user before execution. As shown in Figure 1, task alignment is a subset of broader task discovery, which constitutes resolving the intended task from any source (user or environment).
Why is task alignment important? In user-centric agentic tasks (i.e. Shopping, GDPVal), models must align with users on their intended task before they are able to execute them. On agentic coding tasks, for example, task alignment is the important first step, as recognized in both industrial coding agents like OpenAI’s Codex or Claude Code, and research [1]. Otherwise, failing to align in this first stage can lead to models executing the wrong task, resulting in unexpected outcomes and losses in money and time during downstream execution.
We acknowledge that in a more general task discovery setting with multiple sources of information beyond the user (i.e. a codebase), certain types of missing task attributes may be resolvable through defaults or interacting with the environment. Therefore, we agree resolving task uncertainty during execution would make the evaluation stronger for task discovery. However, given there is little effort in this new direction, this work focuses on task alignment evaluation as a starting point. Modeling partial observability with diverse sources and during execution is the next follow up (i.e., a Pro version of evaluation).
Regarding interaction ability, we ground our interaction claims in our comparison between model and human assistants in the same setting, where human assistants also do not see alternatives or task attributes but nonetheless beat models at interacting with users to align on their tasks. As asked, we also provide below an interaction budget control forcing 14 models to ask for at least 8 turns and compare CommitAcc@1 against human assistants' conversations per-turn, reinforcing that the human advantage holds on a per-turn basis.
Thank you again for your review. We have added a more explicit framing in Section 3, the interaction budget control, and details to environment scope in Limitations.
[1] Humans are Missing from AI Coding Agent Research: Wang et al., arXiv
[2] Grounding in Communication: Clark et al., Perspectives on Socially Shared Cognition