acceptodds
Under review as a conference paper at ICLR 2027

Interactive Task Alignment as a POMDP

Abstract

Current benchmarks for language models primarily evaluate execution on fully specified tasks. However, real user tasks are often ambiguous. Users arrive with incomplete, exploratory, or even inconsistent goals, requiring the assistant to first determine the intended task before carrying it out. We study this problem as task alignment: the ability to align with a user on their intended task. We introduce a general framework for converting specified tasks into underspecified interactions, formalized as a POMDP in which the model must infer a latent task from partial and evolving user intent. We validate our user simulator post hoc with a human user study. Across shopping, coding, and professional work settings, we find that while models often perform well once the task is specified, they still struggle with task alignment: current models act prematurely, interact ineffectively, and fail to resolve ambiguous requests. Models on average recover the user's intended task 16-22% of the time under ambiguity. Compared with humans on a random subset of tasks, humans recover 48% of intended tasks versus a 26% model average, beating all evaluated models. Given the gap, we evaluate two approaches to improving task alignment: (1) belief-state planning, and (2) post-training with supervised fine-tuning and reinforcement learning. We notice while all tested methods can improve task alignment, they still generally lag behind humans, suggesting that current models still lack key interaction abilities required for reliable agency.

Primary area:
datasets and benchmarks
Keywords:
LLM benchmarks, LLM agents, user intent elicitation, human-AI interaction, task underspecification

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_t9qzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates interactive task alignment and propose an evaluation framework. The framework requires models to interact with the user and then evaluates alignment with the user's true intention by candidate ranking.

Critical strengths
  1. The problem is framed well, and the cross-domain evaluation provides a useful perspective beyond conventional task-completion benchmarks.
  2. The human-transcript replay experiment is informative.
Critical weaknesses
  1. The assistant cannot access candidate tasks during interaction but only receives their complete specifications during evaluation. Therefore, the metric combines information elicitation with identification of emerging alternatives. Ranking the correct task first does not necessarily mean that the assistant had formed the correct actionable understanding when it committed.
  2. The evaluation conflates clarification behavior with discovering the relevant task space. If the assistant does not observe the alternative options or explore the environment, it may not be clear which differences need to be clarified. Some perturbed attributes may also represent implementation choices that could reasonably be resolved through defaults or environment inspection. Therefore, low recovery scores and stable stopping tendencies do not uniquely demonstrate deficient interaction policies. Controlled experiments with fixed interaction budgets and access to task dimensions or environment exploration would help distinguish these explanations.
Questions for authors

Please see Critical Weaknesses above.

5 comments

Thank you for the constructive review. We address your concerns below.

1: Validation that assistant actually commits to the recovered task

As in pJFM, we provide support that the recovered task in our setting is the task that the assistant actually commits to and executes. We provide two analyses: (1) first we audit the task spec proposed at the end of the task alignment episode against the rank-1 task, and additionally (2) we have the model execute the task based on the task alignment episode and we measure whether the execution trace aligns with the rank-1 task. We exclude Shopping from these two validations, since Shopping already maps to execution (i.e. selecting an option). In n=140 episodes from GDPVal and Terminal-Bench across n=20 tasks and 3 models selected randomly, we find agreement ranges from 0.75-0.96 with the recovered rank-1 task in both the (1) proposed task spec and (2) the execution trace. We use Deepseek-V4.1 as Judge for measuring agreement.

AnalysisDomainAgreement with rank-1 task
(1) Task spec the assistant proposes at endTerminal-Bench0.96 ± 0.03
(1) Task spec the assistant proposes at endGDPVal0.75 ± 0.05
(2) Execution traceTerminal-Bench0.86 ± 0.05
(2) Execution traceGDPVal0.79 ± 0.04

Furthermore, to validate our judge, we use our judge to evaluate the task spec and execution traces against each of the 14 non-rank-1 tasks, and we find low average agreement scores over all non-rank-1 tasks.

Input to judgeTerminal-BenchGDPVal
Task spec the assistant proposes at the end0.19 ± 0.030.05 ± 0.01
Execution trace0.19 ± 0.030.06 ± 0.01

The two validations for task spec proposal and execution trace are added to the paragraph containing the other task validity checks in lines 229-35, with full details in the Appendix.

2: Clarification behavior and task discovery

We agree that our evaluation jointly measures both clarification behavior and task discovery. We would like to clarify that we follow the established definitions of task alignment in [1][2] where task alignment is defined as resolving the intended task from the user before execution. As shown in Figure 1, task alignment is a subset of broader task discovery, which constitutes resolving the intended task from any source (user or environment).

Why is task alignment important? In user-centric agentic tasks (i.e. Shopping, GDPVal), models must align with users on their intended task before they are able to execute them. On agentic coding tasks, for example, task alignment is the important first step, as recognized in both industrial coding agents like OpenAI’s Codex or Claude Code, and research [1]. Otherwise, failing to align in this first stage can lead to models executing the wrong task, resulting in unexpected outcomes and losses in money and time during downstream execution.

We acknowledge that in a more general task discovery setting with multiple sources of information beyond the user (i.e. a codebase), certain types of missing task attributes may be resolvable through defaults or interacting with the environment. Therefore, we agree resolving task uncertainty during execution would make the evaluation stronger for task discovery. However, given there is little effort in this new direction, this work focuses on task alignment evaluation as a starting point. Modeling partial observability with diverse sources and during execution is the next follow up (i.e., a Pro version of evaluation).

Regarding interaction ability, we ground our interaction claims in our comparison between model and human assistants in the same setting, where human assistants also do not see alternatives or task attributes but nonetheless beat models at interacting with users to align on their tasks. As asked, we also provide below an interaction budget control forcing 14 models to ask for at least 8 turns and compare CommitAcc@1 against human assistants' conversations per-turn, reinforcing that the human advantage holds on a per-turn basis.

TurnsModel, own stoppingModel, forced ≥8 turnsHuman conversation
331.629.832.8
532.535.044.9
832.037.654.6

Thank you again for your review. We have added a more explicit framing in Section 3, the interaction budget control, and details to environment scope in Limitations.

[1] Humans are Missing from AI Coding Agent Research: Wang et al., arXiv

[2] Grounding in Communication: Clark et al., Perspectives on Socially Shared Cognition

Reviewer_t9qzOpenReview ↗

Response to authors

I appreciate the quick response and the additional validation and turn-budget analyses.

The reported agreement rates show how often the final task specification and execution trace match the top-ranked candidate across the audited episodes. Could you also report, separately for GDPVal and Terminal-Bench, how often the final specification and execution trace each match the ground-truth task in episodes where that task was ranked first? This would clarify how reliably a correct ranking reflects what the assistant committed to and attempted to execute.

Your point that the evaluation captures both clarification and task discovery also made me wonder how Section 4.4 draws the line between communication and inference. Suppose a user asks for headphones for running. An assistant that knows about noise-isolating earbuds but is unaware of open-ear headphones might never ask whether the user wants to block out distractions or stay aware of their surroundings. If a gap in world knowledge keeps the assistant from recognizing what to ask, how would you account for that when interpreting the human advantage in Section 4.4?

Thank you for the follow-up questions. We address both below.

1: Agreement with ground-truth task when ranked first

As asked, we report below how often the final specification and execution trace match the rank-1 task on a random subset of n=40 episodes where the ground-truth task was correctly ranked first.

AnalysisDomainAgreement with ground-truth task at rank-1
(1) Task spec the assistant proposes at endTerminal-Bench0.98 ± 0.01
(1) Task spec the assistant proposes at endGDPVal0.91 ± 0.03
(2) Execution traceTerminal-Bench0.95 ± 0.03
(2) Execution traceGDPVal0.89 ± 0.03

Same as with any rank-1 task, we see that when the correct task is ranked first, the assistant also largely commits to and executes the correct task. We will shortly add this to the validity paragraph in lines 229-35, and the Appendix.

2: World knowledge v.s. interaction in Section 4.4

We agree that knowledge in general affects interaction ability, where as in your example knowing about which attributes exist and how they can vary can affect what questions the assistant asks. To account for knowledge, we provide two analyses.

(1) Knowledge control. First we provide a knowledge control prior to any interaction, where for each of the n=15 products in the human assistant study, we give the {human, model} assistant the user’s initial intent, along with the product category, and ask the assistant to list out the attributes relevant to the user’s task. We compare this list against a ground-truth attribute list that we curate manually from the full candidate set with LLM assistance.

(2) Attributes asked in conversation. We also analyze the task alignment transcripts to identify and measure the proportion of ground-truth attributes the assistant asks the user about in the conversation.

We use Deepseek-V-4.1 as Judge for evaluating against the ground-truth attributes in both analyses (1), (2). We test on all 14 models and a set of n=45 new humans different from the original human assistant study. Results are reported below.

AssistantKnowledge controlAttributes asked in conversationAttributes asked, forced ≥8 turns
Human assistants0.23 ± 0.060.53 ± 0.03—
Model assistants0.64 ± 0.030.26 ± 0.020.45 ± 0.03

On the knowledge control, we first see that models recall more relevant task attributes than humans, which is consistent with prior work showing that LLMs exceed non-expert human baselines on general knowledge benchmarks [1][2]. Despite this, humans still cover more task dimensions than models in the conversation (0.53 vs. 0.26 base). Models ask about only 35% of the task dimensions they themselves initially listed, further reinforcing interaction rather than knowledge as the gap.

We note that the human participants in the knowledge control are different from the participants in the human assistant study (i.e. population-level comparison), and that factors such as small sample size and lack of human effort may have resulted in biased underestimates of human knowledge. However, our communication result is nonetheless supported by the observation that models know attributes but do not ask about them. We have updated the interaction framing in Section 4.4 with the role of world knowledge, and added both analyses, with full details in the Appendix.

Thank you again for your thoughtful review. We believe these clarifications and additions have made the paper stronger, and we hope they address your concerns.

[1] Hendrycks et al., Measuring Massive Multitask Language Understanding, ICLR 2021

[2] Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark, COLM 2024.

Reviewer_pJFMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework to evaluate whether LLMs can clarify ambiguous tasks before executing them. The authors convert tasks from Shopping-MMLU, GDPVal, and Terminal-Bench into interactive environments using an LLM user simulator. During the interaction, the assistant does not see the candidate specifications; instead, an independent ranker evaluates whether the target task can be identified from the conversation context. The authors benchmark 15 models, highlighting poor task-recovery performance for underspecified requests, and explore methods like belief-state planning and RL to improve alignment. The main contribution is the evaluation framework and the empirical analysis of task discovery, rather than a novel algorithmic breakthrough.

Critical strengths
  1. A useful evaluation target with a reusable construction Separating task discovery from execution is a great idea: an execution score often hides whether a model simply misunderstood the prompt. Sections 3-4 provide a solid methodology to convert existing benchmarks into interactive alignment tasks. The consistent behavioral differences in when models choose to commit make this framework highly informative.

  2. Complementary experiments make the diagnosis actionable The transcript replay experiment in Section 4.4 is particularly insightful. The fact that models perform better when reading human-led conversations proves that poor interaction—not just poor inference—is the bottleneck. Additionally, the interventions in Section 4.5 show that the benchmark is responsive to alignment tuning.

Critical weaknesses
  1. Correct candidate ranking is not yet validated as correct commitment The evaluation relies on a separate ranking probe that can see the candidate set, whereas the assistant commits without seeing it. While this measures if the task is recoverable from the conversation, it does not prove the assistant actually committed to the right task. An audit of the final commitment messages is needed. Furthermore, there is a discrepancy in the reported planning results: the text and Table 2 claim 39.4% CommitAcc@1 on Shopping for the planner, but Table 11 reports 19.4%.

  2. The intervention comparisons do not isolate the claimed mechanism The planner adds multiple components (hypothesis generation, question proposals, answer modeling) and extends interactions by ~3 turns. There is no matched-turn or matched-compute baseline to show if the improvement comes from the max-belief planning objective itself or just from talking longer. A baseline using Expected Information Gain (EIG), as seen in prior works like BED-LLM, would be a much stronger control since the authors explicitly swapped EIG for max-belief.

Questions for authors
  1. Does the assistant actually commit to the recovered task? What fraction of the final commitment messages actually specify the correct target without relying on the candidate set probe? Please also clarify the discrepancy between Table 2 (reporting 39.4% for Shopping +plan) and Table 11 (reporting 19.4%).

  2. Do planning gains survive matched budgets and a direct EIG control? How much of the planning gain remains if the base policy is forced to interact for the same number of turns? Additionally, please compare the max-belief planner against a standard EIG-based planner (using the same hypothesis/question pipeline) to isolate the benefit of the proposed objective.

  3. How robust is the human and simulator evidence across tasks? Please provide the per-assistant human vs. simulator differences for the human-assistant study. Does the human advantage hold up if we control for conversation length during transcript replay?

5 comments

Thank you for highlighting the usefulness of our evaluation direction. We appreciate the helpful feedback, and we address each of your concerns below.

1: Validation that assistant actually commits to the recovered task

First, we provide support that the recovered task in our setting is the task that the assistant actually commits to and executes. We provide two analyses: (1) first as asked, we audit the task specification proposed at the end of the task alignment episode against the rank-1 task, and additionally (2) we have the model execute the task based on the task alignment episode and we measure whether the execution trace aligns with the rank-1 task. We exclude Shopping from these two validations, since Shopping already maps to execution (i.e. selecting an option). In n=140 episodes from GDPVal and Terminal-Bench across n=20 tasks and 3 models selected randomly, we find agreement ranges from 0.75-0.96 with the recovered rank-1 task in both the (1) proposed task specification and (2) the execution trace. We use Deepseek-V4.1 as Judge for measuring agreement.

AnalysisDomainAgreement with rank-1 task
(1) Task spec proposed at endTerminal-Bench0.96 ± 0.03
(1) Task spec proposed at endGDPVal0.75 ± 0.05
(2) Execution traceTerminal-Bench0.86 ± 0.05
(2) Execution traceGDPVal0.79 ± 0.04

Furthermore, to validate our judge, we use our judge to evaluate the task spec and execution traces against each of the 14 non-rank-1 tasks, and we find low average agreement scores over all non-rank-1 tasks.

Input to judgeTerminal-BenchGDPVal
Task spec proposed at the end0.19 ± 0.030.05 ± 0.01
Execution trace0.19 ± 0.030.06 ± 0.01

On the discrepancy between Table 2 and Table 11, we could not find the 19.4% discrepancy. Both tables report the same planner result for Shopping: 39.4 ± 2.5% CommitAcc@1 ("+plan" column in both) which is the right number. We suspect the confusion may have been from the 19.3% on line 1038. If you were referring to a different location, we would be grateful if you could point us to it!

The two validations for task spec proposal and execution trace are added to the paragraph containing the other task validity checks in lines 229-35, with full details in the Appendix.

2: Planner performance on per-turn basis, and EIG baseline

We provide analysis of planner performance hold on a per-turn basis. Below is the per-turn average CommitAcc@1 across all 14 models on an expanded n=150 human assistant set with and without planning, where we observe an edge starting at turn 3. We also add an turn budget control, where we force base to interact for at least 8 turns before allowing commitment.

TurnBasePlannerBase forced to ≥8 turns
124.522.017.9
331.632.329.8
532.536.835.0
1032.241.838.3
1532.044.038.1
Median base commit turn32.033.028.6

Additionally, we add an EIG baseline to compare against max-belief score for planning. For Qwen 3.5 9B, we see that swapping the question selection score for EIG is slightly better on Shopping and otherwise within noise, suggesting planning gain comes from tracking beliefs explicitly, rather than choice of objective. We observe that 22.6% turns have different questions asked between max-belief and EIG.

TaskMax-beliefEIG
Shopping39.5 ± 2.544.4 ± 2.6
GDPVal21.6 ± 3.520.2 ± 3.4
Terminal-Bench33.8 ± 3.434.3 ± 3.9

3: Human performance on per-turn basis, and simulator evidence

For human and simulator evidence, after expanding our human study to n=186 human users over 4 assistant models, we show that the per-assistant metrics between humans and our simulator largely agree, where model ordering on our simulator matches humans exactly on both CommitAcc@1 (deepseek > 397b > 9b > gpt-5.4) and CommitReward (gpt-5.4 > deepseek > 397b > 9b).

AssistantCommitAcc@1 (Human)CommitAcc@1 (Simulator)CommitReward (Human)CommitReward (Simulator)
GPT-5.424.4 ± 6.827.6 ± 5.91.109 ± 0.0511.094 ± 0.051
DeepSeek-V4.1-Flash52.4 ± 7.841.3 ± 6.71.078 ± 0.0601.030 ± 0.060
Qwen 3.5 397B38.9 ± 6.732.7 ± 5.40.961 ± 0.0621.014 ± 0.045
Qwen 3.5 9B26.5 ± 6.430.6 ± 5.50.879 ± 0.0690.922 ± 0.057

Lastly, we provide per-turn CommitAcc@1 of humans v.s. base models below, showing the human advantage over models starts at turns 3-4 on the human assistant set.

TurnBaseHuman
124.517.5
331.632.8
532.544.9
1032.256.3
1532.057.2
Median base commit turn32.036.5

Thank you again for your detailed review. We believe these additions have made the paper stronger, and we hope they address your concerns.

Dear Reviewer pJFM,

As the discussion period closes on Nov 18, we wanted to check whether our response had addressed your concerns. We would be glad to answer any remaining questions.

Dear Reviewer pJFM,

As it is the last day of the discussion period, we would like to follow up and check whether your concerns have been addressed. We would welcome any further comments or questions.

Reviewer_i4auOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing benchmarks score end-to-end success on underspecified tasks, which mixes up failures of understanding with failures of execution. This paper isolates “task alignment”: how well an assistant works out, through conversation, which task the user actually wants.

Task alignment is set up as a POMDP. The hidden state is the true task plus the user’s current intent. The assistant sees only the conversation and, at each turn, either asks something or commits to a task. An LLM user simulator answers from its current intent, and an “intent evolver” that knows the true task gradually moves that intent from an abstract request toward the full specification. The authors convert Shopping-MMLU, GDPVal and Terminal-Bench this way. Each task gets initial requests at three levels of abstraction and 15 candidate specifications: the true task plus perturbations of it. After each turn, a probe in a separate thread asks the assistant to rank the candidates. The candidates are never shown in the dialogue, and the probe doesn’t affect it. The metrics are top-1 accuracy at commitment, normalized rank, a reward with a turn cost, and information gain.

Most models recover the true task only a small fraction of the time at the abstract tier, and models that commit early resolve less uncertainty. In a human study on five Shopping tasks, human assistants beat all 15 models. Models score much higher when ranking from the human transcripts, which points to communication rather than inference. A second study replaces the simulator with real users for three assistants. Finally, for Qwen3.5-9B, belief-state planning greatly improves results on Shopping, while SFT and RL help less.

Critical strengths
  1. Existing benchmarks mix failures of understanding with failures of execution. This framework isolates understanding, and it works on any benchmark with specified tasks, so it can be reused across domains. The approach is therefore extendable to other benchmarks.
  2. The authors design the probes very carefully. The assistant never sees the candidates during the dialogue, so it cannot read the answer off them. An empty-conversation test is near chance, which shows the target cannot be guessed from the candidates alone.
  3. The paper runs both a human-as-assistant study and a human-as-user study, which is rare for LLM-simulator evaluations. Models do much better on human transcripts, which suggests that communication drives much of the gap.
Critical weaknesses
  1. In the main text, humans and models are compared only on top-1 accuracy, on a small subset of Shopping tasks at the concrete tier. Human turn counts are not reported anywhere. Table 4 in the appendix shows that humans have the lowest turn-penalized reward of all assistants (0.64 vs. 0.70–1.10). So it is unclear whether humans did better because they understood the user better or because they asked more questions. The human advantage may not hold once turns are penalized. The result may also depend on which tasks were sampled: Table 3 shows very large differences across the five items.
  2. The authors do not fully show that the simulated user behaves like real users. The comparison was made after the simulator was finalized, and only on Shopping tasks where the user starts with a detailed request, while the main results use vague starting requests and other domains. The main text reports only averages pooled across the three assistants. The per-assistant results in Table 5 show that the simulator does not keep the real-user ordering. On CommitAcc@1, real users rank GPT-5.4 > Qwen-397B > Qwen-9B, while the simulator ranks them Qwen-397B > Qwen-9B > GPT-5.4. Please discuss this in the main text.
  3. The evidence that SFT and RL improve alignment is mostly from one domain (Shopping). In the other two domains the held-out sets are small and the gains are modest, and SFT lowers accuracy in one of them. Planning gains appear in all three domains, but it also makes conversations longer, so part of its gain may come from more turns rather than from better questions.
  4. Each model was run once per task, at temperature zero with reasoning off or at its lowest setting. The paper does not show that the ranking of models would stay the same on repeat runs or with reasoning turned on.
Questions for authors
  1. Can you report human turn counts, explain why humans have the lowest CommitReward in Table 4, and add more tasks to the human study?
  2. The comparison between real and simulated users covers only a few assistants. Can you extend it to more assistants and check whether the simulator preserves their ordering, both with vague starting requests and on a different domain?
  3. Can you compare humans and models at the same number of turns? Can you also run repeat runs to show that the model ranking is stable?
  4. Does belief-state planning still beat the base model at an equal number of turns and LLM calls?
  5. Can you evaluate SFT and RL on larger held-out sets in the other two domains?
5 comments

Thank you for highlighting the reusability of our framework and care in our setting design. We address each of your raised concerns below.

1: Human turn counts, CommitReward, and humans and planner at equal turns (W1, W3, Q1, Q3, Q4)

First as asked, we expand the human assistant study to n=150 human assistant, increasing the number of tasks from 5 to 15. As below, our results on the gap between humans > models all hold with greater power under the expanded dataset. We also report human turn counts.

SourceCommitAcc@1CommitRewardMean turns
Humans51.0 ± 5.61.029 ± 0.0538.3 ± 0.3
Models (mean over 14)32.0 ± 2.20.912 ± 0.0373.7 ± 0.4

Regarding the low CommitReward for humans in Table 4, we investigated and found that human CommitReward was computed on a different scale than models due to a typo. We appreciate the catch, and have now corrected this in Table 4 where we use the same reward parameters for models and humans. With the corrected scale, humans score 1.044 rather than 0.642, where humans outperform almost all models in terms of CommitReward on the n=50 assistant set, which aligns with the trends in CommitAcc@1 and also the expanded n=150 human assistant set.

Next, we provide per-turn CommitAcc@1 of humans and the planner v.s. base models across 14 models on the expanded human assistant dataset. To control for interaction budget, we also add a control where we force the base model to interact for at least 8 turns (which is the matched median turn count of humans/planner).

TurnBaseHumanPlannerBase forced to ≥8 turns
124.517.522.017.9
228.022.325.524.0
331.632.832.329.8
431.039.433.731.0
532.544.936.835.0
731.752.340.537.9
1032.256.341.838.3
1532.057.244.038.1
Median base commit turn32.036.533.028.6

Comparing at the same number of turns, we see that human transcripts still lead by +16.9 at turn 8 for all 14 models, reinforcing that the gap is in what models ask and not only in how long they ask. For the planner, we see a smaller advantage over base models, where the planner remains ahead of the forced base at equal turns (+5.9 ± 3.2).

Regarding the LLM calls required for belief-state planning, we acknowledge that this approach incurs additional calls per turn to generate hypotheses, propose candidate questions, and update explicit beliefs, and is therefore not compute-matched with the base model. We report the number of calls and tokens per turn in the Appendix and discuss the associated computational overhead in the Limitations section.

We would like to emphasize that the primary goal of this paper is to evaluate the strengths and trade-offs of different approaches to resolving uncertainty in task alignment. For example, explicitly tracking and reasoning over task beliefs can improve alignment performance, but may come at the cost of computational efficiency.

2: Expanded simulator evidence, and model ordering (W2, Q2)

As in pJFM, we also expand our human user study to n=186 human users over 4 assistant models, demonstrating that the per-assistant metrics between humans and our simulator largely agree, where model ordering on our simulator matches humans exactly on both CommitAcc@1 (deepseek > 397b > 9b > gpt-5.4) and CommitReward (gpt-5.4 > deepseek > 397b > 9b).

AssistantCommitAcc@1 (Human)CommitAcc@1 (Simulator)CommitReward (Human)CommitReward (Simulator)
GPT-5.424.4 ± 6.827.6 ± 5.91.109 ± 0.0511.094 ± 0.051
DeepSeek-V4.1-Flash52.4 ± 7.841.3 ± 6.71.078 ± 0.0601.030 ± 0.060
Qwen 3.5 397B38.9 ± 6.732.7 ± 5.40.961 ± 0.0621.014 ± 0.045
Qwen 3.5 9B26.5 ± 6.430.6 ± 5.50.879 ± 0.0690.922 ± 0.057

We will update the main text with the expanded user simulator validation, along with the intent evolver ablation in the Appendix.

Regarding user studies with more task ambiguity and in other domains beyond Shopping, we acknowledge both of these concerns in our Limitations section, where (1) more genuine task uncertainty is hard to capture while requiring a ground-truth, and (2) finding industry experts at the level of GDPVal and Terminal-Bench is both difficult and costly. We leave user studies in these directions to future work, which we fully intend to pursue.

3: Stability across runs, and reasoning (W4, Q3)

For stability across runs, we run each of the 14 models three times on all GDPVal and Terminal-Bench ) tasks, and compare model rankings across the three runs. We observe that model rankings largely agree across runs, with Kendall's correlation of 0.83 (GDPVal) and 0.90 (Terminal-Bench) for CommitAcc@1, and respectively 0.66 and 0.81 for CommitReward.

Regarding reasoning, we run each model at its lowest reasoning setting so that all models are compared apples-apples. This was also done in consideration of response latency for our user studies. Our intuition is that more reasoning likely does impact performance, as suggested by the increased performance in planning, but we do not explore this dimension due to computational cost estimates of additional $5-10K.

4: SFT and RL on non-Shopping domains (W3, Q5)

First, we note that training in GDPVal and Terminal-Bench domains were limited by the size of the source benchmarks. As a result, the training and held-out sets were both small in size at around n=40, which led to possible undertraining and limited evaluation. We highlight the scalability of our Shopping domain as an advantage, where we notice that we can scale task alignment training and transfer gains to other domains, indicating task alignment is a missing general capability more than task-specific skill dependent on domains .

However, in the rebuttal, we still test whether training in GDPVal and Terminal-Bench domains generalizes to larger held-out set. Here are the evaluation results of Qwen3.5-9B checkpoints trained on GDPVal and Terminal-Bench on the larger Shopping held-out set,

ModelCommitAcc@1Mean turnsΔ vs base
Base19.3 ± 2.08.1—
GDPVal SFT11.8 ± 1.63.6−7.5 ± 2.3
GDPVal RL24.0 ± 2.28.3+4.6 ± 2.4
Terminal-Bench SFT13.6 ± 1.74.3−5.7 ± 2.3
Terminal-Bench RL19.3 ± 2.08.4+0.0 ± 2.4

Likely due to limited training data as explained earlier, SFT checkpoints underperform on Shopping, while RL checkpoints actually maintain performance, with GDPVal RL improving CommitAcc@1 from 19.3% to 24.0%.

Thank you again for your detailed review. We will shortly add these additions to make the paper stronger, and we hope they address your concerns.

Dear Reviewer i4au,

As the discussion period closes on Nov 18, we wanted to check whether our response had addressed your questions. We would be happy to clarify anything else.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.