- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EA-Bench, a benchmark for evaluating how efficiently agents handle missing, redundant, or conflicting user instructions. It includes 270 conditions across 18 task families and measures both unnecessary tool calls and information requests, while distinguishing correct answers from those supported by sufficient information. Experiments on nine models show that imperfect instructions reduce accuracy and increase execution costs, even when agents complete tasks correctly.
Critical strengths
- Varying instruction quality while keeping tasks and data fixed helps isolate the effects of missing, redundant, and conflicting information.
- The benchmark distinguishes correct outputs from success supported by sufficient information, and measures overhead against necessary actions and information needs. This provides more insight than accuracy alone.
Critical weaknesses
- The 270 conditions come from only 18 underlying tasks, without validation across different data, policy values, or repeated runs, limiting generalizability.
- Information acquisition and scoring depend on a user simulator and automatic annotation. Counting annotation errors as failures may conflate agent limitations with evaluation-system errors.
- Efficiency is mainly measured on successful attempts. The shared-success comparison across all nine models includes only 20 tasks—and just 17 for information analysis—making efficiency rankings difficult to establish reliably.
Questions for authors
- How often does the user simulator reject reasonable questions or give incomplete answers? Could the authors manually check a sample to see how much this affects the results?
- The 847.5% increase in information requests seems partly expected, since more information is missing. Given that average excess requests actually decrease, is this evidence of inefficient questioning?
- The efficiency comparison uses a small set of tasks that all models solve successfully. Do the rankings still hold when failed attempts and their costs are included?
4 comments
Part II
**W2. Information acquisition and scoring depend on a user simulator and automatic annotation. **
The revised version replaces Qwen3-8B with GPT-5.5 (medium effort) for user simulation and artifact grading. Predefined task knowledge and authoritative facts constrain the simulator’s responses.
For identified service or response issues, we rerun the corresponding tasks after recovery. For artifact-grading issues, we preserve the original artifacts and regrade them. The revised evaluation uses the outcomes obtained after recovery.
Incomplete information records do not automatically cause task failure. The current 2,088 evaluation records contain 888 successful and 1,200 failed outcomes. The failure distribution is:
| Failure category | Easy | Hard | Total |
|---|---|---|---|
| Incorrect result | 379 | — | 379 |
| Missing prerequisite steps | 34 | — | 34 |
| No usable final answer | 37 | 190 | 227 |
| Incorrect or missing files | — | 89 | 89 |
| Incorrect or incomplete content | — | 471 | 471 |
| Total failures | 450 | 750 | 1,200 |
Furthermore, 40 runs with incomplete fact-request records are still graded as successful, demonstrating that information-record completeness and task outcomes are handled separately. We use grounded credit to examine correct outputs obtained without sufficient supporting information. Among 635 successful Easy runs under imperfect conditions, 144 (22.7%) receive ungrounded credit, showing that outcome correctness does not necessarily imply acquisition of the information needed to support that outcome.
W3. Efficiency is mainly measured on successful attempts. The shared-success comparison across all nine models includes only 20 tasks—and just 17 for information analysis—making efficiency rankings difficult to establish reliably.
The revised tool-call and user-interaction statistics include successful and failed runs, while fact-request statistics use runs with complete records. Figure 5 further separates successful and failed runs when reporting efficiency gaps.
We also provide a supplementary comparison on identical condition sets, requiring valid records and references for every model without filtering for shared success:
| Metric | Shared base tasks | Shared conditions | Failed runs included |
|---|---|---|---|
| 18 | 68 | Yes | |
| 32 | 100 | Yes |
This ensures that models are compared on the same conditions within each metric. Efficiency analysis also examines each model’s execution behavior, since unsuccessful runs can consume substantial resources. For example, Gemini 3.5 Flash exhausts the 100-tool-call budget in all 30 Hard Full conditions, yet fails to fully complete any task.
Section 5 and Appendix A report each model’s tool-call counts (Figure 3), requested fact counts (Figure 4), and user-interaction counts (Figure 11) across instruction conditions. Figure 12 further compares costs between successful and failed runs.
Q1. How often does the simulator reject reasonable questions or provide incomplete answers? Has a sample been manually checked?
We acknowledge the importance of manual auditing for assessing simulator reliability. The revised version uses a stronger simulator with explicit knowledge boundaries and separately records requested facts and facts actually returned. Only returned facts count as acquired information; an unanswered request does not count as successful information acquisition.
Human experts have screened tasks and variants for realism, but this does not constitute an independent reliability audit of simulator interactions. We do not yet have human-verified estimates of reasonable-question rejection rates, incomplete-answer rates, or their effects on model results. The simulator upgrade and structured records improve traceability, while these error rates require targeted manual review of interaction records.
Part III
Q2. Does the 847.5% increase in information requests establish inefficient questioning?
We agree that this increase includes the cost of recovering necessary information and cannot directly establish inefficient questioning. We have therefore removed “information requests increase by 847.5%” from the abstract and separately report actual costs and gaps relative to task-specific reference requirements.
The Easy cost results from Figures 3–4 are summarized below:
| Actual cost metric | Full | Imperfect conditions |
|---|---|---|
| Mean tool calls | 10.73 | 14.03 |
| Mean requested facts | 0.028 | 1.393 |
Following the main-text aggregation, we first average perturbation-column means within each model and then average equally across the nine models.
The results still show that imperfect information increases actual execution and information-acquisition costs. Tool-call counts increase by 14.4%–71.5% across models on Easy, while mean fact requests increase from approximately 0.0 to 1.4. Meanwhile, 22.7% of successful Easy outputs under imperfect conditions receive ungrounded credit, further demonstrating the need to evaluate task completion, information grounding, and execution costs separately.
Q3. What does the efficiency comparison show when failed runs are included?
We recovered and retried identified infrastructure-error cases and updated the evaluation on the expanded task set following AI-assisted checks and expert screening. The supplementary results below use identical condition sets across models and include both successful and failed runs:
| Model | Mean | Mean |
|---|---|---|
| Astra | +0.12 | +0.11 |
| Kimi K3 | +0.13 | −0.49 |
| HY4 | +0.24 | −0.34 |
| GPT-5.5 | −0.24 | −0.57 |
| Qwen3.8 Max | +0.00 | −0.66 |
| Seed 2.1 Pro | +0.03 | −0.64 |
| Gemini 3.5 Flash | +0.25 | −0.53 |
| GLM-5.3 | +0.22 | −0.48 |
| DeepSeek V4 Pro | −0.53 | −0.54 |
uses 68 shared Easy conditions, and uses 100 shared Easy conditions. † Includes supplemental-solver executions.
With failed runs included, negative gaps indicate fewer recorded operations or fact requests than the reference requires. They may accompany premature output, incomplete execution, or failure to acquire necessary information. These values therefore require interpretation alongside task outcomes and grounded credit; lower gaps do not automatically indicate greater efficiency. Hard runs that exhaust the call budget without completing the task are examined separately through actual costs and failure distributions.
Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.
Dear Reviewer,
As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.
We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.
Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.
Thank you for your review, and we always look forward to your response.
Part I
We thank the reviewer for recognizing our controlled paired design and the value of evaluating task correctness, grounding, and efficiency together. We appreciate the opportunity to clarify the concerns raised and respond below.
We expanded the benchmark from 18 to 63 base-task instances. Building on the original 18 closed-state tasks, we added 15 application-operation tasks and 30 professional workflow tasks, covering scenarios from JobBench, AppWorld-UL, and WorkBench. Appendix C provides the complete DAG abstractions and node descriptions for all base tasks.
While preserving the original task objectives and solvability, we organize workflow operations, information sources, and prerequisite dependencies into DAGs. Counterfactual analysis identifies facts that affect task decisions, enabling the construction of Missing, Redundant, Conflict, and Mixed variants.
We also introduce a two-stage checking process. First, AI-assisted checks assess whether missing facts remain inferable, whether conflicting statements have a clear authority ordering, and whether redundant information preserves the original facts and task requirements. These are combined with counterfactual and executable checks to verify variant validity. Human experts then screen variants for realistic user communication scenarios. Ultimately, we retain 232 evaluation conditions from 792 candidates, comprising 63 Full baselines and 169 imperfect variants.
The revised benchmark divides tasks into Easy and Hard tracks according to their inherent structure and deliverable requirements:
Easy primarily covers application operations and structured office workflows. Hard tasks are constructed from JobBench and involve databases, multi-file analysis, cross-source information integration, and complex artifact generation. For example, JB17 requires analyzing 3,930 ITSM ticket records, combining asset inventories and project budgets, and producing a four-sheet workbook and a gap-register CSV. JB11 involves 20 applicants and 1,470 employee records, with Word and Excel deliverables. These additions broaden the diversity of business records, task structures, and deliverable requirements.
Statistically, we pair variants with their corresponding Full conditions and aggregate with equal base-task weights. Appendix A also reports 95% intervals from 5,000 source-cluster bootstrap resamples, preserving relationships among related observations.