acceptodds
Under review as a conference paper at ICLR 2027

Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent

Abstract

Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent. However, this oracle communication assumption rarely holds in practice: users may miscommunicate, change their goals, and run out of patience. We define this task setting as interactive intent alignment, where agents must recover and continuously track the user's current intent despite imperfect communication and evolving goals. To study this setting, we introduce Drift-Bench++, a principled benchmark construction pipeline for verified executable tasks with controlled misalignment and intent shifts, along with an interaction protocol featuring finite patience, diverse simulated users, and silent interaction-conditioned shifts. We further develop GRIP, a comprehensive evaluation protocol covering task grounding, user realism, inquiry effectiveness, and adaptation to evolving intent. Across diverse environments, models, and interaction conditions, stronger interaction consistently helps but remains far from oracle performance; Validation on deployed ProdAgent sessions further shows that the modeled failures are prevalent and consequential in deployment. By providing a unified, executable benchmark for interactive intent alignment, Drift-Bench++ offers a foundation for evaluating and advancing agents under realistic communication and evolving intent.

Primary area:
datasets and benchmarks
Keywords:
Interactive Intent Alignment, Simulated User Agentic RL, Intent Shifting, Communication Misalignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_m6UhOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on interactive intent alignment, and proposes Drift-Bench++, a benchmark construction pipeline and evaluation framework, GRIP, for assessing LLM agents under the aforementioned focus. This benchmark is built upon three existing tool-using benchmarks, and the proposed pipeline extracts structured intent graphs containing frame and conditions, edges spanning refinement, relaxation, substitution, and pivot relations.

Then GRIP is used to evaluate the outcoming resutls, decoupling the final outcomes into interaction-oriented measurements. Experiments evaluate seven clarification baselines and three reference agents across five LLM backbones.

Critical strengths
  • Verifiable data construction pipeline. The motivation and data construction process are clearly stated, and aligned.

  • Comprehensive diagnostic evaluation protocol. GRIP provides isolated analysis on different intent elicitation (Earned vs. Inferred), inquiry targeting (Aim vs. Recovery), and adaptation dynamics under non-stationary goals (Staleness, PostShift Success), which is informative and could be valuable for furture improvement.

Critical weaknesses
  • The contribution of this paper is much overstated, and the novelty argument understates both cited and omitted prior work. More fatally, some conclusion of cited papers are incorrect.

    • Runtime interaction limits already exist: Tables 1 and 3 mark prior benchmarks as lacking "Patience-bounded" interaction. However, ChatShop (Section 4 and Appendix B.3) limits turn budgets and rejects further requests; BIRD-INTERACT (Section 4) explicitly incorporates user patience parameters and depleted action budgets; and EvolIF [1] (Section 3.3) terminates execution upon patience exhaustion.
    • Ambiguity and changing goals have already been combined: AppWorld-UL (Section 4.1) composes underspecification with solicited revision of infeasible goals; TRIP-Bench (Section 3.2) evaluates ambiguous intent shifts elicited after clarification; and ToolHaystack [2] (Section 4.2) explicitly distinguishes implicit from explicit shifts. Asserting that all prior shifts are explicitly announced (line 165-168 in this paper) oversimplifies existing work, and overstates the scope and contribution of this paper.
    • The evaluation and construction precedents also require carefully examination. Evolving Intent (Table 3) directly evaluates turn-wise intent tracking, despite its negative beyond-success flag in this paper. Important omitted comparisons include [3] for verified nonideal-communication construction, [4] for conversation-conditioned hidden-intent refinement, and [5] for practical belief/profile reconciliation during interaction.
  • Overclaimed scope: the mismatch between general claims and tool-use limitation.

    • This paper frames "Interactive Intent Alignment" as a fundamental, universal task setting for human-agent interaction (line 38-53). However, the benchmark operationalization in Drift-Bench++ is strictly restricted to structured, executable tool-use environments. This paper does propose valuable thoughts, yet its implementation does not align with its motivation (which is also a "Interactive Misalignment" that authors claimed to solve a broader issue but insteadily narrow down to a small part during the method design and experiments).

[1] Jia, Qi, et al. "One battle after another: Probing llms’ limits on multi-turn instruction following with a benchmark evolving framework." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[2] Kwak, Beong-woo, et al. "ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions." EMNLP (Findings). 2025.

[3] Yang, Xuan, et al. "Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions." arXiv preprint arXiv:2606.03318 (2026).

[4] Dai, Andy, et al. "Interactive Task Alignment as a POMDP." arXiv preprint arXiv:2607.16412 (2026).

[5] Ruan, Minyuan, et al. "Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind." arXiv preprint arXiv:2602.13832 (2026).

Questions for authors
  1. After accounting for the overlapping capabilities in prior benchmarks, what specific capability or evaluation insight does Drift-Bench++ add? Please revise the comparison tables to reflect that distinction consistently.

  2. Real-world interactive miscommunication frequently arises in non-executable, open-ended tasks where goals are fuzzy, non-extensional, or preference-based. Can the 3-step verification and GRIP evaluation protocol generalize to these non-tool domains without defaulting back to unverified LLM-as-a-judge evaluators?

  3. Could the agent actually know the goal it is judged against? How much of the reported failure comes from agents mishandling discoverable goal changes, and how much comes from changes they cannot observe before committing? Please quantify these cases

Reviewer_WaoJOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Drift-Bench ++, a benchmark for evaluating interactive intent alignment in LLM agents. It also propose GRIP, a comprehensive evaluation protocol covering task grounding, user realism, inquiry effectiveness, and adaption to evolving intent. Experiment across multiple benchmark environment shows interactive clarification improves agent performance. This was also validated on real-world setting.

Critical strengths

Key motivation seems convincing and the benchmark seems to be able to address the problems from motivation.

Wide range of experiments and ablation studies looks sound.

Using actual production data makes this work more meaningful.

Critical weaknesses

Large portion of the setting seems to be similar with last year ICLR paper “Non-Collaborative User Simulators for Tool Agents”. Key motivation is pretty similar and some of the parts such as “contextual irrelevance”, “Insufficient information”, “Patience-bounded interaction” are already used in the above paper. However, I couldn’t find any justification about the contribution of the paper compare to the above paper.

Questions for authors

I ask the authors to revise the manuscript(especially introduction and existing challenges) based on additional research of related paper.

Intent shifting seems like a bit rule based transition. Some of this transition may be unnatural making the benchmark unreliable. Can you check the actual task trajectories and give a justification of this part.

Minor editorial comments : Table caption should go top of the table not under.

Reviewer_aUFLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the idealized assumptions of existing intent-recognition benchmarks, which typically presume that users communicate their needs completely, maintain fixed intentions, and explicitly announce any changes. It introduces Drift-Bench++, a new evaluation framework for interactive intent alignment. The framework establishes a systematic paradigm for data perturbation and intent construction, using executable intent graphs, controlled communication failures, finite user patience, and silent intent shifts to simulate realistic interactions in which user requests may be inaccurate and goals may evolve over time. Building on this framework, the paper systematically evaluates whether current models can recognize when interaction is necessary, effectively clarify missing information, and continuously maintain, update, and verify the user’s current intent. Through multidimensional metrics, diverse interaction conditions, and real-world production sessions, it further examines whether the benchmark reflects the distribution of intent misalignment and shifting observed in practice, supported by a comprehensive set of experiments.

Critical strengths
  1. This work introduces a new evaluation paradigm for interactive intent alignment, supported by a complete data construction pipeline and comprehensive evaluation metrics.
  2. The experiments provide valuable insights into model performance under more realistic intent-recognition scenarios involving imperfect communication and evolving user intent. The appendix provides extensive experiments across different methods and model combinations, offering strong support for the main conclusions.
  3. The paper is well written, clearly structured, and easy to follow.
Critical weaknesses
  1. I was wondering whether the authors could further explore a patience mechanism that changes with interaction length or time—for example, gradually increasing the penalty for repeated clarification questions—to better reflect how user patience may decline over the course of an interaction.
  2. I would appreciate clarification on whether the generated benchmark samples were manually inspected to assess the correctness and naturalness of the perturbations. In addition, while the current production-data analysis shows that these communication failures and intent shifts do occur in real-world settings, it may be helpful to further explore how closely the benchmark aligns with real data in terms of category frequencies, language use, and interaction patterns.
Questions for authors

Same as above

Reviewer_Cw8yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous works' problems: do not jointly consider non-oracle communication and intent shifting, do not model user patience well. This paper claims to address all three by

  1. first generate graphs of interconnected intents, then apply miscommunication.
  2. weighted user patience, under partial control by persona. The paper evaluates various harnesses, llms and compare against real human interaction results.
Critical strengths

Good transparency: website, code and generated data immediately available.

Critical weaknesses

Critique:

  1. There seems to be a contradiction. Does Oracle Communication Assumption imply a fixed intent? line 012 seems to say this but not line 048 or Table 1.
  2. User simulation.
    • How are the patience budgets and deductions fitted to real humans?
    • The regenerated intent graph always has a depth-one star as subgraph. This limits the possible topology of generated graphs.
    • Later intent shifts indeed create difficulty for the agent to handle, but the miscommunication perturbation only acts at the beginning of the episode and doesn't inherently perturb later user responses, even when still on the root node. Thus, although initial miscommunication can interact with later shifts, later miscommunication is not explicitly modeled. This restriction isn't addressed by either answers to RQ2 or the Limitations section. Figure 7 does not reveal the position of the miscommunication inside each episode either.
      • Does the average user display such a behavioral tendency? If the ability to serve these intended users displaying such a behavioral pattern can be evaluated by first measure under miscommunication, then separately under intent shift, the paper's motivation for joint evaluation would be weakened. What effects are revealed in the combined setting that separation would miss?
    • For identifying misalignment in ProdAgent, it seems that user-side annotation is relied upon. Since true intentions are not directly accessible, is there mechanism to differentiate reported miscommunication corrections from intent shifts?
Questions for authors
  1. Table 4 partially reveals verifier rejection rates, but we would like to see the effects of all verifiers individually; specifically, the denominators and rejection rates. This would clarify how useful each verifier is, and effects on the final benchmark.
Reviewer_zHk9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Drift-Bench++, a benchmark for evaluating agents under miscommunication, evolving user intent, and finite patience. It combines verified communication faults, silent intent shifts, and persona-conditioned users with the GRIP evaluation framework. Experiments demonstrate the benefits of clarification and persistent alignment challenges, while production-session analysis supports the benchmark’s real-world relevance.

Critical strengths
  1. The authors introduce Drift-Bench++, a principled pipeline for constructing verified executable tasks with controlled misalignment and intent shifts, providing a systematic testbed for evaluating agents’ proactive clarification and intent-tracking capabilities.
  2. The proposed GRIP protocol goes beyond task success to assess task grounding, simulated-user fidelity, clarification effectiveness, and adaptation to evolving intent, enabling a more comprehensive diagnosis of interactive alignment.
Critical weaknesses
  1. The evaluation is limited to WebShop and τ²-bench, which involve relatively constrained user interactions and environments. The applicability to more complex agent tasks, such as ALE, remains unverified. Additional experiments and analysis are needed to demonstrate how perturbations can be introduced in these settings
  2. Both perturbation generation and simulated-user responses rely on LLMs. Despite the verification pipeline and state-machine controls, the paper would benefit from a quantitative assessment of how faithfully generated utterances reflect the intended perturbations and authorized disclosures. Residual simulator errors could affect the evaluation metrics and should be analyzed.
Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.