- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes XYEval, a benchmark for evaluating the XY problem in the agent setting, where the user may provide a misleading solution alongside the problem description. The benchmark mutates six benchmarks, including HLE, SWE-Bench Pro, Terminal Bench, MCP-Atlas, SWE-Verifier, and -Bench, showing that all frontier models are susceptible to the XY problem. They also show that it's challenging to mitigate the XY problem, which requires the agent to recognize the misdirection and communicate clearly.
Critical strengths
- The problem setup is interesting; it offers a new perspective to evaluate the sycophancy of LLMs.
- The benchmark and evaluation are comprehensive, covering mainstream agentic benchmarks and frontier models, with ablation studies like pedantic users, mitigation baselines, and failure mode analysis.
- The paper writing is easy to follow.
Critical weaknesses
- Although not 100% overlap, it seems like there is a bunch of work that studies how misspecification will affect the LLM's answer to the problem (e.g., https://arxiv.org/abs/2504.14119). The author mentioned the differences in lines 113- 114, but I still feel the XY setup is just a different prompt template; fundamentally, it's still the misspecification problem.
- Some of the experimental setup is vague/unclear. For example, it's unclear why Gemini3.7-flash has an outlier where, after adding XY-instruction, the accuracy even improved. I have no idea whether it is due to randomness or can be reproduced across multiple random seeds.
Questions for authors
- When generating misleading suggestions, are we using Gemini 3.5 Flash for all the benchmarks, or just for coding benchmarks? If we only use it to generate misleading suggestions for coding benchmarks, what model are we using to generate the misleading suggestions for the rest of the benchmarks?
- In the pendantic user setting, how do we implement it? Are we injecting the misleading suggestions in each round's user prompt?
- In the failure mode analysis, the author mentioned that the most common inappropriate compliance type is lack of reasoning. I am curious whether the XY drop can be mitigated by increasing the reasoning effort?
2 comments
I thank the authors for the detailed explanation. Most of my concerns are addressed. On the novelty of the project scope, I'm not fully convinced and still think it largely overlaps with the existing misspecification-related work. I therefore maintain my assessment.
Response to Reviewer U6Cq
We thank the reviewer for recognizing the importance of the XY problem and the breadth of our evaluation.
A1: We agree that XY problems are closely related to instruction misspecification. However, unlike CodeCrash's focus on misleading context in code reasoning, XYEval provides a general meta-evaluation framework for agentic execution, including tool use, multi-turn interactions, and explicit communication of disagreement. Our contribution is the transferable evaluation framework and behavioral analyses, not merely a new prompt template.
In XYEval, we also explicitly frame the suggestions as “I think”... phrasings in the user suggestions to preserve the intactness of the original task/environment. This also means that agents can falsify the misleading suggestion by running tests, calling tools, and checking against given policy.
A2: Good observation! We cannot fully rule out noise in sampling, and will note this limitation. We evaluated models with a single sample and sometimes on benchmark subsets mainly to prioritize studying the XY issue in multiple domains/benchmarks under budget constraint.
Questions
A3: SWE-bench uses Gemini 3.5 Flash; Terminal-Bench uses each Gemini model itself and Gemini 3.5 Flash for Claude/GPT; and τ²-bench uses expert-designed rules. We will highlight these more in the appendix.
A4: The pedantic instructions are applied to the user simulator across interaction turns, requiring the agent to explain why the misleading suggestion is inappropriate. Agent-side defense instructions are also maintained across turns (Sec. 5.2).
A5: Great question! We used the default maximum thinking level so that we can reproduce the baseline numbers. We will highlight this direction for future work.