- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper focuses on interactive intent alignment, and proposes Drift-Bench++, a benchmark construction pipeline and evaluation framework, GRIP, for assessing LLM agents under the aforementioned focus. This benchmark is built upon three existing tool-using benchmarks, and the proposed pipeline extracts structured intent graphs containing frame and conditions, edges spanning refinement, relaxation, substitution, and pivot relations.
Then GRIP is used to evaluate the outcoming resutls, decoupling the final outcomes into interaction-oriented measurements. Experiments evaluate seven clarification baselines and three reference agents across five LLM backbones.
Critical strengths
-
Verifiable data construction pipeline. The motivation and data construction process are clearly stated, and aligned.
-
Comprehensive diagnostic evaluation protocol. GRIP provides isolated analysis on different intent elicitation (Earned vs. Inferred), inquiry targeting (Aim vs. Recovery), and adaptation dynamics under non-stationary goals (Staleness, PostShift Success), which is informative and could be valuable for furture improvement.
Critical weaknesses
-
The contribution of this paper is much overstated, and the novelty argument understates both cited and omitted prior work. More fatally, some conclusion of cited papers are incorrect.
- Runtime interaction limits already exist: Tables 1 and 3 mark prior benchmarks as lacking "Patience-bounded" interaction. However, ChatShop (Section 4 and Appendix B.3) limits turn budgets and rejects further requests; BIRD-INTERACT (Section 4) explicitly incorporates user patience parameters and depleted action budgets; and EvolIF [1] (Section 3.3) terminates execution upon patience exhaustion.
- Ambiguity and changing goals have already been combined: AppWorld-UL (Section 4.1) composes underspecification with solicited revision of infeasible goals; TRIP-Bench (Section 3.2) evaluates ambiguous intent shifts elicited after clarification; and ToolHaystack [2] (Section 4.2) explicitly distinguishes implicit from explicit shifts. Asserting that all prior shifts are explicitly announced (line 165-168 in this paper) oversimplifies existing work, and overstates the scope and contribution of this paper.
- The evaluation and construction precedents also require carefully examination. Evolving Intent (Table 3) directly evaluates turn-wise intent tracking, despite its negative beyond-success flag in this paper. Important omitted comparisons include [3] for verified nonideal-communication construction, [4] for conversation-conditioned hidden-intent refinement, and [5] for practical belief/profile reconciliation during interaction.
-
Overclaimed scope: the mismatch between general claims and tool-use limitation.
- This paper frames "Interactive Intent Alignment" as a fundamental, universal task setting for human-agent interaction (line 38-53). However, the benchmark operationalization in Drift-Bench++ is strictly restricted to structured, executable tool-use environments. This paper does propose valuable thoughts, yet its implementation does not align with its motivation (which is also a "Interactive Misalignment" that authors claimed to solve a broader issue but insteadily narrow down to a small part during the method design and experiments).
[1] Jia, Qi, et al. "One battle after another: Probing llms’ limits on multi-turn instruction following with a benchmark evolving framework." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[2] Kwak, Beong-woo, et al. "ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions." EMNLP (Findings). 2025.
[3] Yang, Xuan, et al. "Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions." arXiv preprint arXiv:2606.03318 (2026).
[4] Dai, Andy, et al. "Interactive Task Alignment as a POMDP." arXiv preprint arXiv:2607.16412 (2026).
[5] Ruan, Minyuan, et al. "Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind." arXiv preprint arXiv:2602.13832 (2026).
Questions for authors
-
After accounting for the overlapping capabilities in prior benchmarks, what specific capability or evaluation insight does Drift-Bench++ add? Please revise the comparison tables to reflect that distinction consistently.
-
Real-world interactive miscommunication frequently arises in non-executable, open-ended tasks where goals are fuzzy, non-extensional, or preference-based. Can the 3-step verification and GRIP evaluation protocol generalize to these non-tool domains without defaulting back to unverified LLM-as-a-judge evaluators?
-
Could the agent actually know the goal it is judged against? How much of the reported failure comes from agents mishandling discoverable goal changes, and how much comes from changes they cannot observe before committing? Please quantify these cases