acceptodds
Under review as a conference paper at ICLR 2027

EA-Bench: Measuring information and action efficiency under imperfect instructions

Abstract

In real-world settings, users often provide incomplete, distracting, or inconsistent information. A useful agent must recognize these imperfections and resolve task-relevant uncertainty through efficient, targeted questioning. Measuring this ability requires separating execution difficulty from the quality of the instruction. To make this distinction measurable, we introduce the Efficient Agent Benchmark (EA-Bench). It holds task records and authoritative policies fixed while systematically varying missing information, redundancy, and conflicting instructions. EA-Bench comprises 270 paired conditions across 18 manually selected synthetic task families. We use task dependency graphs to enforce multi-step execution and counterfactual policy worlds to identify necessary clarifications, creating challenging imperfect instructions consistent with user intent. Experiments on nine mainstream open- and closed-weight model services show that combining multiple imperfections reduces pooled task accuracy from 76.5% to 60.2%. Across missing and mixed conditions, the hardest cases achieve accuracies of only 5.6% and 22.2%. More seriously, even when agents still complete tasks correctly under imperfect information, they make 23.6% more tool calls, request 847.5% more information items, and take 41.0% longer to complete the tasks.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, LLM, Agent Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_dicDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EA-Bench, a benchmark for evaluating how efficiently agents handle missing, redundant, or conflicting user instructions. It includes 270 conditions across 18 task families and measures both unnecessary tool calls and information requests, while distinguishing correct answers from those supported by sufficient information. Experiments on nine models show that imperfect instructions reduce accuracy and increase execution costs, even when agents complete tasks correctly.

Critical strengths
  1. Varying instruction quality while keeping tasks and data fixed helps isolate the effects of missing, redundant, and conflicting information.
  2. The benchmark distinguishes correct outputs from success supported by sufficient information, and measures overhead against necessary actions and information needs. This provides more insight than accuracy alone.
Critical weaknesses
  1. The 270 conditions come from only 18 underlying tasks, without validation across different data, policy values, or repeated runs, limiting generalizability.
  2. Information acquisition and scoring depend on a user simulator and automatic annotation. Counting annotation errors as failures may conflate agent limitations with evaluation-system errors.
  3. Efficiency is mainly measured on successful attempts. The shared-success comparison across all nine models includes only 20 tasks—and just 17 for information analysis—making efficiency rankings difficult to establish reliably.
Questions for authors
  1. How often does the user simulator reject reasonable questions or give incomplete answers? Could the authors manually check a sample to see how much this affects the results?
  2. The 847.5% increase in information requests seems partly expected, since more information is missing. Given that average excess requests actually decrease, is this evidence of inefficient questioning?
  3. The efficiency comparison uses a small set of tasks that all models solve successfully. Do the rankings still hold when failed attempts and their costs are included?
4 comments

We thank the reviewer for recognizing our controlled paired design and the value of evaluating task correctness, grounding, and efficiency together. We appreciate the opportunity to clarify the concerns raised and respond below.

W1. The 270 conditions come from only 18 underlying tasks, without validation across different data, policy values, or repeated runs, limiting generalizability.

We expanded the benchmark from 18 to 63 base-task instances. Building on the original 18 closed-state tasks, we added 15 application-operation tasks and 30 professional workflow tasks, covering scenarios from JobBench, AppWorld-UL, and WorkBench. Appendix C provides the complete DAG abstractions and node descriptions for all base tasks.

While preserving the original task objectives and solvability, we organize workflow operations, information sources, and prerequisite dependencies into DAGs. Counterfactual analysis identifies facts that affect task decisions, enabling the construction of Missing, Redundant, Conflict, and Mixed variants.

We also introduce a two-stage checking process. First, AI-assisted checks assess whether missing facts remain inferable, whether conflicting statements have a clear authority ordering, and whether redundant information preserves the original facts and task requirements. These are combined with counterfactual and executable checks to verify variant validity. Human experts then screen variants for realistic user communication scenarios. Ultimately, we retain 232 evaluation conditions from 792 candidates, comprising 63 Full baselines and 169 imperfect variants.

The revised benchmark divides tasks into Easy and Hard tracks according to their inherent structure and deliverable requirements:

TrackBase tasksWorkflow DAG nodesFullMissingRedundantConflictMixedTotal
Easy334–133345144114147
Hard3013–303010319585

Easy primarily covers application operations and structured office workflows. Hard tasks are constructed from JobBench and involve databases, multi-file analysis, cross-source information integration, and complex artifact generation. For example, JB17 requires analyzing 3,930 ITSM ticket records, combining asset inventories and project budgets, and producing a four-sheet workbook and a gap-register CSV. JB11 involves 20 applicants and 1,470 employee records, with Word and Excel deliverables. These additions broaden the diversity of business records, task structures, and deliverable requirements.

Statistically, we pair variants with their corresponding Full conditions and aggregate with equal base-task weights. Appendix A also reports 95% intervals from 5,000 source-cluster bootstrap resamples, preserving relationships among related observations.

TrackPerturbationBase tasks / source clustersConditionsScore difference95% interval
EasyMissing29 / 2845−19.16[−27.78, −10.92]
EasyRedundant12 / 1214+1.85[−5.56, +10.19]
EasyConflict32 / 3141−14.93[−22.74, −6.94]
EasyMixed9 / 914−27.57[−41.36, −14.20]
HardMissing7 / 710+3.16[−3.28, +10.24]
HardRedundant25 / 2531+0.93[−3.42, +5.27]
HardConflict8 / 89+3.44[−2.38, +9.74]
HardMixed5 / 55+7.35[−3.45, +17.93]
Differences are measured in score points relative to the corresponding Full baselines.

**W2. Information acquisition and scoring depend on a user simulator and automatic annotation. **

The revised version replaces Qwen3-8B with GPT-5.5 (medium effort) for user simulation and artifact grading. Predefined task knowledge and authoritative facts constrain the simulator’s responses.

For identified service or response issues, we rerun the corresponding tasks after recovery. For artifact-grading issues, we preserve the original artifacts and regrade them. The revised evaluation uses the outcomes obtained after recovery.

Incomplete information records do not automatically cause task failure. The current 2,088 evaluation records contain 888 successful and 1,200 failed outcomes. The failure distribution is:

Failure categoryEasyHardTotal
Incorrect result379—379
Missing prerequisite steps34—34
No usable final answer37190227
Incorrect or missing files—8989
Incorrect or incomplete content—471471
Total failures4507501,200

Furthermore, 40 runs with incomplete fact-request records are still graded as successful, demonstrating that information-record completeness and task outcomes are handled separately. We use grounded credit to examine correct outputs obtained without sufficient supporting information. Among 635 successful Easy runs under imperfect conditions, 144 (22.7%) receive ungrounded credit, showing that outcome correctness does not necessarily imply acquisition of the information needed to support that outcome.

W3. Efficiency is mainly measured on successful attempts. The shared-success comparison across all nine models includes only 20 tasks—and just 17 for information analysis—making efficiency rankings difficult to establish reliably.

The revised tool-call and user-interaction statistics include successful and failed runs, while fact-request statistics use runs with complete records. Figure 5 further separates successful and failed runs when reporting efficiency gaps.

We also provide a supplementary comparison on identical condition sets, requiring valid records and references for every model without filtering for shared success:

MetricShared base tasksShared conditionsFailed runs included
EtE_t1868Yes
EiE_i32100Yes

This ensures that models are compared on the same conditions within each metric. Efficiency analysis also examines each model’s execution behavior, since unsuccessful runs can consume substantial resources. For example, Gemini 3.5 Flash exhausts the 100-tool-call budget in all 30 Hard Full conditions, yet fails to fully complete any task.

Section 5 and Appendix A report each model’s tool-call counts (Figure 3), requested fact counts (Figure 4), and user-interaction counts (Figure 11) across instruction conditions. Figure 12 further compares costs between successful and failed runs.

Q1. How often does the simulator reject reasonable questions or provide incomplete answers? Has a sample been manually checked?

We acknowledge the importance of manual auditing for assessing simulator reliability. The revised version uses a stronger simulator with explicit knowledge boundaries and separately records requested facts and facts actually returned. Only returned facts count as acquired information; an unanswered request does not count as successful information acquisition.

Human experts have screened tasks and variants for realism, but this does not constitute an independent reliability audit of simulator interactions. We do not yet have human-verified estimates of reasonable-question rejection rates, incomplete-answer rates, or their effects on model results. The simulator upgrade and structured records improve traceability, while these error rates require targeted manual review of interaction records.

Q2. Does the 847.5% increase in information requests establish inefficient questioning?

We agree that this increase includes the cost of recovering necessary information and cannot directly establish inefficient questioning. We have therefore removed “information requests increase by 847.5%” from the abstract and separately report actual costs and gaps relative to task-specific reference requirements.

The Easy cost results from Figures 3–4 are summarized below:

Actual cost metricFullImperfect conditions
Mean tool calls10.7314.03
Mean requested facts0.0281.393

Following the main-text aggregation, we first average perturbation-column means within each model and then average equally across the nine models.

The results still show that imperfect information increases actual execution and information-acquisition costs. Tool-call counts increase by 14.4%–71.5% across models on Easy, while mean fact requests increase from approximately 0.0 to 1.4. Meanwhile, 22.7% of successful Easy outputs under imperfect conditions receive ungrounded credit, further demonstrating the need to evaluate task completion, information grounding, and execution costs separately.

Q3. What does the efficiency comparison show when failed runs are included?

We recovered and retried identified infrastructure-error cases and updated the evaluation on the expanded task set following AI-assisted checks and expert screening. The supplementary results below use identical condition sets across models and include both successful and failed runs:

ModelMean EtE_tMean EiE_i
Astra+0.12+0.11
Kimi K3+0.13−0.49
HY4+0.24−0.34
GPT-5.5−0.24−0.57
Qwen3.8 Max+0.00−0.66
Seed 2.1 Pro+0.03−0.64
Gemini 3.5 Flash+0.25−0.53
GLM-5.3+0.22−0.48
DeepSeek V4 Pro−0.53−0.54

EtE_t uses 68 shared Easy conditions, and EiE_i uses 100 shared Easy conditions. † Includes supplemental-solver executions.

With failed runs included, negative gaps indicate fewer recorded operations or fact requests than the reference requires. They may accompany premature output, incomplete execution, or failure to acquire necessary information. These values therefore require interpretation alongside task outcomes and grounded credit; lower gaps do not automatically indicate greater efficiency. Hard runs that exhaust the call budget without completing the task are examined separately through actual costs and failure distributions.

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Reviewer_Yc7KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EA-Bench, a benchmark designed to evaluate both the task accuracy and execution/information efficiency of LLM agents under imperfect user instructions. Across 18 synthetic task families adapted from prior benchmarks the authors enforce multi-step tool execution via operational dependency graphs (DAGs) and define necessary user clarifications via counterfactual policy worlds. This formalization provides explicit task-level lower bounds for both tool actions and information requests. Furthermore, the paper distinguishes "raw outcome success" from "grounded success" (where all required missing/conflicting facts were explicitly acquired). Experiments across nine open- and closed-weight models reveal that instruction imperfections significantly degrade accuracy, expose frequent "ungrounded" lucky guesses, and lead to substantial overhead in tool calls and user questioning even when tasks succeed.

Critical strengths
  1. Separating "grounded success" from lucky guesses is a very important contribution. In many existing agent benchmarks, if an agent happens to output the right final state, it gets full credit even if it ignored an ambiguity. By constructing counterfactual policy worlds (Proposition 2), EA-Bench strictly defines which hidden facts are mathematically necessary to guarantee correctness. The finding in Section 6.1 that models frequently achieve raw correctness without acquiring necessary information (e.g., Qwen3.8-Max only has 22 grounded successes out of 41 correct outputs in Missing conditions) is a strong result that shows why outcome evaluation alone is insufficient.

  2. The benchmark provides clean, explicit lower bounds (A∗A^* and I∗I^*) to jointly measure action and information efficiency. By keeping the underlying records fixed and enforcing execution order through opaque handles in a DAG, the authors isolate the exact overhead caused by instruction imperfections. This reveals a very interesting trade-off in Table 3: some models (like GPT-6-Astra) are extremely efficient with tool calls but over-ask redundant confirmation questions even in the Full condition, whereas others (like Qwen3.8-Max) ask questions selectively but waste steps in tool execution.

Critical weaknesses
  1. Evaluation system and annotator errors are counted directly as agent failures. In Sections 3.3 and 5.2, the setup relies on a relatively small model (Qwen3-8B) for both user simulation (scope stage) and post-run atomic fact annotation. However, the protocol states that simulator scope errors and annotation errors count as task failures in the main accuracy metric (Table 2, Figure 8). Also, as shown in Appendix I.4, when the simulator itself over-generates and restates already-known policy facts, the trajectory is flagged as needs_review and dropped from strict information efficiency calculation. It seems unfair to penalize the evaluated models' accuracy or filter out their valid runs due to flaws in the 8B simulator/annotator pipeline.

  2. The action lower bound A∗=K+1[I∗>0]A^* = K + 1[I^* > 0] in Equation (2) makes an overly strong assumption about batching questions. It assumes that whenever I∗>0I^* > 0, an optimal agent can always ask for all missing or conflicting facts in a single ask_user call. While this holds if all policy gaps are obvious from the initial prompt, in multi-step workflows an agent might reasonably need to call read_* tools to inspect the actual data records before realizing that two rules conflict in practice. Asking follow-up clarification questions across two turns should not necessarily be penalized as suboptimal action overhead.

Questions for authors
  1. In Figure 8, several models have a noticeable number of "Runtime error" and "Annotation error" outcomes, which are counted as failures in Table 2. Could you break down how many of these were caused by the evaluated agent itself (e.g., invalid JSON) versus failures in the Qwen3-8B simulator/annotator or API transport? How would the main accuracy table look if infrastructure/annotation errors were excluded or re-run?

  2. Regarding the 1[I∗>0]1[I^* > 0] bound: in your 18 task families, are all missing and conflicting policy requirements identifiable purely from reading the initial prompt, or do some conflicts only become visible after calling read_* tools on the task records?

  3. Clarification / future work on dataset diversity: Although there are 270 conditions, they are built on 18 synthetic task families where all 15 variants within a family share the exact same underlying business records and policy values (Appendix B). Have you tested how sensitive the model behaviors are if you vary the underlying numerical records or data snapshots within each family?

3 comments

We thank the reviewer for recognizing our distinction between grounded success and lucky guesses, and our joint evaluation of action and information efficiency. We appreciate the opportunity to clarify the concerns raised and respond below.

W1. Evaluation system and annotator errors are counted directly as agent failures.

We agree that simulator, annotator, or API transport errors should not be directly interpreted as limitations of the evaluated agent. The revised version replaces Qwen3-8B with GPT-5.5 (medium effort) for user simulation and artifact grading, with predefined knowledge boundaries constraining the simulator’s responses.

For identified service or response issues, we rerun the corresponding tasks after recovery. For artifact-grading issues, we preserve the original artifacts and regrade them. The revised evaluation uses the outcomes obtained after recovery.

We also separate task-outcome grading from information-record completeness. Runs with incomplete fact-request records retain their task outcomes but are excluded from statistics requiring complete request records. Among the current 2,088 evaluation records, 78 have incomplete fact-request records, and 40 of these are still graded as successful. Incomplete information records therefore do not automatically cause task failure and are not treated as zero requests.

W2. The action lower bound A∗A^* in Equation (2) makes an overly strong assumption about batching questions

We thank the reviewer for highlighting that information gaps may emerge progressively during multi-step execution. We clarify how our dependency structure, information accounting, and action reference address this issue below.

  • The DAG contains multiple dependency levels, allowing information gaps to emerge during execution.

    During task construction and expert screening, we consider how imperfect information is distributed across dependency levels. Agents may need to use read_* tools to inspect actual records before identifying information gaps or conflicts affecting subsequent decisions. As illustrated in Figure 2, different nodes depend on different files. Appendix C provides the DAGs and node descriptions for all base tasks.

  • Requested facts and user interactions are counted separately.

    EiE_i counts requested atomic facts. Asking for two facts in one turn or requesting them separately across two turns yields the same count of two, provided neither fact is requested repeatedly. Splitting clarification across turns therefore does not itself increase EiE_i. Figure 4 reports fact-request counts, while Figure 11 separately reports user-interaction counts.

  • The action reference applies when questions can be batched.

    Required reading and processing operations are included in KK. When the necessary facts can be jointly identified and requested after these operations, the corresponding action reference is $ A^{\mathrm{ref}}=K+\mathbf{1}[I^{\mathrm{ref}}>0]. $

    The call costs of multi-turn clarification are recorded by EtE_t and analyzed alongside task outcomes. For tasks where a subsequent question can only be identified after receiving an earlier answer, the attainability of this reference requires further examination.

Q1. What do the evaluation results look like after infrastructure-error recovery?

We have completed recovery processing for the identified infrastructure-error cases and updated the corresponding evaluation results. The revised overall results are summarized below:

ModelEasy Total / GHard Total / G
Astra82.3 / 81.655.9 / 55.9
Kimi K377.6 / 63.313.2 / 12.4
HY476.9 / 68.010.8 / 10.5
GPT-5.568.0 / 54.415.0 / 14.2
Qwen3.8 Max65.3 / 51.712.2 / 11.4
Seed 2.1 Pro63.9 / 49.012.6 / 12.2
Gemini 3.5 Flash59.2 / 44.20.0 / 0.0
GLM-5.354.4 / 46.312.0 / 11.2
DeepSeek V4 Pro46.3 / 37.412.6 / 11.8
G denotes grounded score. Easy reports task pass rates (%); Hard reports mean rubric scores on a 0–100 scale.

Q2. Can all missing or conflicting information be identified from the initial prompt?

Relevant information may be distributed across the initial instruction and task files accessed through tools. As illustrated in Figure 2, different DAG nodes depend on facts from different files. For example, role assignment requires combining role requirements, candidate information, and relevant constraints. Agents need to inspect these materials to determine whether the available information is sufficient.

We therefore do not assume that agents can identify every issue solely from the initial prompt. Necessary read_* calls are part of the execution workflow and count toward action costs. The single-batched-question reference assumes that the relevant questions can be jointly identified and requested after the necessary reading operations.

The complete DAGs provide a basis for examining this assumption task by task. Appendix C presents the complete DAGs and node descriptions for all 63 base tasks, explicitly documenting required operations, information sources, and prerequisite dependencies.

Q3. How has task and data diversity been expanded?

The revised benchmark expands from 18 to 63 base-task instances, adapted from JobBench, AppWorld-UL, and WorkBench. We retain 232 conditions from 792 candidates:

TrackBase tasksFull conditionsImperfect conditionsTotal
Easy3333114147
Hard30305585

Easy primarily covers application operations and structured office workflows, with 4–13 nodes per workflow DAG. The added Hard track contains 30 base tasks constructed from JobBench, with 13–30 DAG nodes, involving databases, multi-file analysis, information integration across sources, and complex artifact generation.

For example, JB17 requires analyzing 3,930 ITSM ticket records, combining asset inventories and project budgets, and producing a four-sheet workbook and a gap-register CSV. JB11 involves 20 applicants and 1,470 employee records, with Word and Excel deliverables. These tasks broaden coverage in business-record scale, data types, dependency structures, and deliverable requirements.

The added tasks also pose substantial challenges to current models. For example, Gemini 3.5 Flash reaches the 100-tool-call budget limit in all 30 Hard Full conditions, yet fails to fully complete any of these tasks.

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Reviewer_UH6zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a benchmark on a request given by the user. Based on how incomplete information is given from the user, the AI finds it hard to navigate the golden answer. This benchmark measures how accurate the answer was, and how many times it bothered the user, and how much more tool calls it generated and more.

Critical strengths

None

Critical weaknesses
  1. Counting questions and tool calls does not capture the user's experience. The paper treats every question beyond a task-specific minimum as "excess" and reports it as user burden. However, whether a question is a burden depends on the human user, and no human was involved at any stage. The user is simulated by Qwen3-8B, and the questions are counted by a post-run annotation model. For example, in the procurement task (Appendix I.2), GPT-6-Astra asks about Gold certification and also re-confirms three conditions already stated in the instruction. The benchmark counts these as three wasted requests, but a real user might see them as reasonable double-checking. The paper itself cites evidence that LLM-simulated users are unreliable proxies for humans (Seshadri et al.), yet it provides no human study or human–simulator agreement to support its burden metric.

  2. The paper motivates efficient recovery but offers no way toward it. The paper frames the goal as agents that recognize information gaps and resolve them through "efficient, targeted questioning," and the conclusion calls for agents that "better recognize when clarification is needed." However, the paper does not investigate how this could be achieved. There is no prompting strategy, training method, or simple baseline intervention showing that the measured inefficiency can be reduced. As a result, the findings stay at the level of "current models are inefficient" and give limited actionable insight.

  3. The paper does not explain its own findings. The results describe what models do but rarely why, which makes it hard to judge whether the differences are meaningful.

  • GPT-6-Astra makes the most unnecessary requests (0.94 per task, Table 3b). It does so even under complete instructions (1.71), where there is nothing to clarify, and it requests 32 facts against a minimum of 11 on shared tasks (Figure 3). At the same time, it has the lowest tool overhead (0.13) and the most information-sufficient answers in the missing-information condition (52/52). The paper does not discuss whether Astra's behavior is a failure or a deliberate confirmation habit that contributes to its reliability. Appendix F.1 acknowledges that broad confirmation "may [be] a strategy for managing an unreliable user channel," and that "controlled experiments would be needed to identify the reason." Those experiments are not provided.
  • The paper reports that higher missingness or conflict levels sometimes improve scores (Section 6.3). It offers only one speculative sentence for this and does not test it.
  • Because the benchmark relies only on counts, it cannot tell apart an agent that struggles from one that is simply cautious. Both appear as "excess."
  1. The headline numbers are presented in a misleading way. The abstract reports that information requests increase by 847.5%. Section 6.2 explains that this comes from 40 versus 379 items, that required information also increases, and that mean excess requests actually decrease. Most of the increase is therefore necessary clarification, not inefficiency, but the abstract presents it as overhead. In addition, each of the 270 conditions is run once per model, and many table cells contain only 18–36 samples, so differences of a few points may not be reliable.
Questions for authors

Why did you use tau bench as comparison? In order to measure the efficiency and helpfulness of an AI, a human involvement plays an important role in scoring matrix. Have you considered involving human annotation or using human data? and why did you think that it will make a benchmark without one?

7 comments

W1: Counting questions and tool calls does not capture the user's experience.

We thank the reviewer for highlighting the distinction between information-request metrics and actual user experience. We agree that users’ perceptions of clarification, reconfirmation, and interaction frequency depend on task risk, individual preferences, and context. We therefore clarify the purpose of the user simulator and the meaning and scope of our efficiency metrics.

  • 1. Simulated users provide a controlled and reproducible environment for evaluating interactions.

    Language-model-based user simulation has been adopted by several interactive-agent benchmarks:

    BenchmarkUser-simulation design
    τ-bench (Yao et al., 2024)Uses language models to simulate users interacting dynamically with agents equipped with domain-specific tools and policy constraints.
    τ²-bench (Barres et al., 2025)Closely couples the user simulator with the environment, constraining its behavior through available tools and observable states.
    AppWorld-UL (Chen et al., 2026)Defines explicit knowledge boundaries for simulated users to support interactions such as clarification and confirmation.

    EA-Bench follows this evaluation approach, using predefined task facts and user knowledge to examine whether agents acquire the information needed to complete tasks.

  • 2. The revised version upgrades the simulator and refines information-request accounting.

    We replace Qwen3-8B with GPT-5.5 (medium effort) and adopt AppWorld-UL’s knowledge-bounded design. The simulator answers relevant questions using predefined task knowledge and returns structured fact identifiers and values.

    We separately record the facts requested by the agent, the facts actually returned by the simulator, and the number of user interactions. This distinguishes multiple fact requests within one interaction, repeated requests across turns, and requests that receive no answer.

  • 3. The procurement example measures repeated requests for information.

    In the procurement example raised by the reviewer, a single agent query contains four requests:

    Requested informationInitial information statusInterpretation
    Whether Gold certification is mandatoryUnspecifiedAcquiring a missing fact needed for the decision
    Whether phased delivery is requiredAlready provided in the instructionReconfirming an available fact
    Whether freight is charged once per supplierAlready provided in the instructionReconfirming an available fact
    Whether installation is charged separatelyAlready provided in the instructionReconfirming an available fact

    Under this task’s information reference:

    Q=4,Iref=1,Ei=3.Q=4,\qquad I^{\mathrm{ref}}=1,\qquad E_i=3.

    Here, Ei=3E_i=3 denotes three fact-request items beyond the information reference requirement. We acknowledge the reviewer’s point that users may welcome reconfirmation in consequential settings such as procurement. However, these confirmations supply no new information needed to complete the current task. From the perspective of information-acquisition efficiency, they are unnecessary and are consequently counted as additional information-request costs. Information efficiency is a core evaluation dimension of EA-Bench; the potential value of these confirmations for building trust, reducing perceived risk, or improving user experience requires separate evaluation.

  • 4. EA-Bench focuses on action and information efficiency under imperfect instructions.

    Our metrics measure actual tool calls, requested fact items, and their differences from task-specific reference requirements. To our knowledge, EA-Bench is the first benchmark to jointly evaluate tool-call and information-request efficiency against task-specific references under controlled missing, redundant, and conflicting instructions. We analyze these costs alongside task completion and grounded credit, focusing on how agents acquire necessary information and execute tasks efficiently. Whether users feel interrupted, prefer reconfirmation, or gain trust through these interactions falls outside the scope of this paper.

W2: The paper motivates efficient recovery but offers no way toward it.

We thank the reviewer for this suggestion. We clarify the scope and contribution of EA-Bench below.

  • 1. EA-Bench aims to measure information recovery and execution efficiency. Developing and validating a new prompting strategy or training method falls outside the scope of this benchmark paper.

    EA-Bench is a benchmark whose primary objective is to systematically measure how imperfect information affects agents’ task completion, information acquisition, and execution efficiency. It provides controlled missing, redundant, and conflicting instruction variants, together with task-specific references for analyzing tool-call and information-request costs. Our contribution is to establish the tasks, metrics, and diagnostic framework needed to evaluate whether such interventions improve information recovery and execution efficiency while preserving task correctness and grounding.

  • 2. The experimental analysis of EA-Bench also identifies typical failure modes, which provide concrete diagnostic targets for improving questioning strategies.

    In Section 5 (Experimental Results), we analyze failures using execution, clarification, and grading records. Figure 8 presents the outcome distribution under imperfect conditions, while Figure 13 examines whether relevant facts were acquired in failed or not fully grounded outputs. The failure distribution is summarized below:

    TrackFailure categoryCountShare of failures within track
    EasyIncorrect result33284.9%
    EasyMissing prerequisite steps266.6%
    EasyNo usable final answer338.4%
    HardIncorrect or incomplete content30362.7%
    HardIncorrect or missing files6814.1%
    HardNo usable final answer11223.2%

    The table aggregates the nine reported model rows and excludes Full conditions. The denominators are 391 Easy failures and 483 Hard failures.

    We further identify two recurring behaviors through the examples in Boxes 1–2:

    • Acting on unresolved information without clarification. Box 1 shows GPT-5.5, Astra, and HY4 making definitive decisions without confirming task-relevant information. Their corresponding Full runs pass, whereas the imperfect-condition runs fail relevant checks. These examples identify the need to recognize unresolved dependencies and obtain clarification before committing to consequential decisions.

    • Repeatedly requesting facts already available. Box 2 shows Astra, HY4, and Kimi K3 asking users to reconfirm information already supplied in task files. Some of these tasks pass but incur avoidable information requests. These examples identify the need to use available evidence before seeking additional clarification.

W3-1: Does Astra’s repeated confirmation reflect a failure or a strategy that supports reliability?

We thank the reviewer for highlighting this distinction. Astra’s low tool overhead and repeated information requests describe different aspects of its behavior. A single user interaction can contain several fact requests, allowing an agent to remain close to the action reference while requesting more information than the information reference requires. We now report tool calls, requested facts, and user interactions separately in Figures 3, 4, and 11, and show the corresponding reference gaps in Figure 5.

We acknowledge that repeated confirmation may be a cautious strategy, particularly in consequential tasks. We therefore interpret these observations as additional information requests relative to the task reference. Task completion and grounded credit are reported separately to capture whether the agent produces a correct, information-supported result.

Comparisons on the same Easy conditions also show that other models can achieve correct and fully grounded results without requesting additional facts:

ConditionAstra’s fact requestsOther models achieving correct, fully grounded results with zero fact requests
AW14/R217
WB01/Full16
WB12/Full18
WB12/R118

For example, in AW14/R2, Astra asks again about credit information already provided, while seven other models complete the same task successfully with full grounded credit and zero fact requests.

W3-2: Why can higher missingness or conflict levels sometimes coincide with improved scores?

Perturbation levels can contain different base tasks and affect different facts. The position of a fact within the dependency graph also matters: one unresolved upstream fact may affect several downstream decisions, while several unresolved facts elsewhere may have a smaller effect. Consequently, the number of missing or conflicting facts alone does not define a uniform difficulty scale.

In the revised analysis, each imperfect condition is compared with the Full condition of the same model and base task. We aggregate variants within each base task and then weight base tasks equally. Figure 6 reports these score changes, Figure 7 reports tool-call changes, and Table 2 documents the task and condition composition.

Figure 6 shows several sequences in which larger perturbations accompany greater grounded-score degradation. For example, the score loss increases from 27.1 points at M=1 to 41.6 points at M=3, and from 9.3 points at C=2 to 25.9 points at C=4. The changes relative to task-matched Full baselines are summarized below:

Perturbation typeLevel 1Level 2Level 3Level 4
Missing−27.1−32.4−41.6−11.1
Conflict−22.2−9.3−19.1−25.9
Redundant+2.5+3.2−0.5—

Redundant conditions show substantially smaller score changes, ranging from −0.5 to +3.2 points. Redundancy preserves the authoritative facts needed to complete the task while adding material for the agent to inspect. We therefore examine its implications for execution costs separately in Figure 7, alongside its comparatively limited effect on task scores.

W3-3: Can the benchmark distinguish an agent that struggles from one that is cautious?

Counts quantify execution and information-request costs, but they do not reveal an agent’s motivation. We therefore present four complementary comparisons: task completion (Table 3 and Figure 8), grounded credit (Table 3), tool-call efficiency (Figures 3 and 5), and user-interaction efficiency (Figures 4, 5, and 11).

The revised analysis in Section 5 uses Figure 8 to categorize outcomes and Figure 13 to examine information acquisition in failed or not fully grounded outputs. Boxes 1–2 provide concrete examples of agents acting on unresolved information and agents reconfirming facts already available.

We believe that a cautious agent may produce a correct, grounded result while incurring additional information-request costs. We therefore evaluate task performance and efficiency separately. A positive efficiency gap does not automatically imply poor task performance or poor user experience.

Reviewer_pxL4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EA-Bench, which measures the task accuracy and efficiency of LLMs under imperfect conditions, including missing information, redundancy, and conflicting instructions. EA-Bench deliberately constructed these conditions from existing benchmarks to probe LLM behavior under imperfect conditions, and the final benchmark size is 270 conditions across 18 synthetic task, each with 15 variants. EA-Bench uses task success metrics, efficiency metrics to measure LLM behaviors and finds that correct task outputs may hide information gaps and that efficient tool use does not necessarily imply selective information acquisition.

Critical strengths
  • The paper discusses an important gap in current LLM evaluation benchmarks. Beyond task completion, efficient execution and grounded execution are important and less studied area.
  • EA-Bench is constructed in a systematic and controlled way. The paired experimental design enables controlled comparisons across different imperfect conditions.
Critical weaknesses
  • The proposed efficiency metrics rely on a predefined execution DAG and a attainable action minimum. In practice, a task may admit multiple correct execution paths with different resource costs. It is unclear whether this reference adequately captures the diversity and variability of valid agent execution strategies. Was the perfect baseline execution evaluated repeatedly to characterize its average resource consumption and variability? How sensitive are the reported efficiency results to alternative valid execution strategies or task decompositions?

  • Table 3, the efficiency results, are computed over model-specific successful task subsets which may differ substantially in difficulty and composition, making cross-model efficiency comparisons potentially misleading. The paper did have a shared-success analysis, but it covers only 20 tasks for action efficiency and 17 for information efficiency, representing a relatively small subset of the benchmark. It primarily reflects the results on the selection of commonly solvable tasks.

  • A more detailed analysis of why agents frequently produce correct outputs without explicitly acquiring the required information would provide more in-depth insights in Section 6.1

Questions for authors

Please see Weakness session.

4 comments

We thank the reviewer for the thoughtful assessment of our DAG-based efficiency evaluation framework and for highlighting the importance of accommodating diverse valid execution strategies. We address the three questions below.

W1 Q1: Does the execution reference accommodate the diversity of valid execution strategies?

We thank the reviewer for raising this question. The DAG describes prerequisite dependencies and permits different execution orders that satisfy them.

When constructing tasks, we organize each workflow as a hierarchical DAG: nodes represent required operations or intermediate artifacts, and edges represent prerequisites for subsequent operations. Its purpose is to identify the information and operations needed to complete the task. Agents can choose execution orders that satisfy these dependencies, and independent operations can be performed in different orders.

As illustrated in Figure 2, different nodes depend on information from different files. For example, role assignment requires combining role requirements, candidate information, and relevant constraints, while subsequent calculations and report generation depend on earlier results. This structure allows us to identify facts that affect specific decisions and construct missing or conflicting variants accordingly.

Task completion is evaluated using the final environment state, submitted files, and task-specific checks or rubrics. Different forms of expression, artifact organization, and execution paths can all receive credit when they satisfy the task requirements. Grounded credit further assesses whether credited outcomes are supported by information the agent actually acquired.

W1 Q2: Was the reference execution evaluated repeatedly to characterize its average resource consumption and variability?

The current reference values are determined from necessary operation dependencies and finite counterfactual analysis. Their calculation does not depend on repeated sampling of an “ideal agent.” Consequently, determining the reference requirements under the current protocol does not require repeated executions to estimate a mean.

These reference values characterize protocol-specific requirements. Characterizing the resource-consumption distributions and variability of different valid strategies is a separate research question beyond the scope of this evaluation.

W1 Q3: How sensitive are the efficiency results to alternative valid execution strategies or task decompositions?

Efficiency metrics are calculated from each actual execution and its corresponding reference values. For each run, we record the actual number of tool calls, AA, and requested facts, QQ. These observed costs capture differences in tool use and information requests resulting from the agent’s choice of execution strategy.

For closed-state tasks, the executor checks dependencies among required operations. We denote the number of required operations by KK and, under a protocol that permits batching requests for necessary facts, define:

Aref=K+1[Iref>0].A^{\mathrm{ref}}=K+\mathbf{1}[I^{\mathrm{ref}}>0].

Here, IrefI^{\mathrm{ref}} is determined by the information needed to distinguish the correct task outcome across finite counterfactual worlds. Its value may differ from the initial number of missing or conflicting facts. The efficiency gaps are defined as:

Et=A−Aref,Ei=Q−Iref.E_t=A-A^{\mathrm{ref}},\qquad E_i=Q-I^{\mathrm{ref}}.

Under a fixed task protocol, differences in actual costs across strategies are directly reflected in EtE_t and EiE_i. Changes to tool granularity or task decomposition require the corresponding reference values to be recomputed. Therefore, the reported efficiency results apply to the current task and interaction protocols; we have not systematically evaluated sensitivity to alternative task decompositions.

By separately evaluating task outcomes, information grounding, and execution costs, we accommodate diverse valid solutions while analyzing the resources consumed by different execution processes.

W2: The paper did have a shared-success analysis, but it covers only 20 tasks for action efficiency and 17 for information efficiency.

We thank the reviewer for pointing out the selection bias introduced by successful-task subsets. We agree that the tasks successfully completed by different models may differ in difficulty and composition, so comparing efficiency only within each model’s successful subset can confound efficiency with task selection. The original small shared-success analysis also primarily reflects tasks that all models can solve. In the revised version, we adjust the statistical scope and jointly analyze task completion, actual costs, and information grounding.

  • Revision 1: Expanding the scope of the main results to include successful and failed runs.

    The revised overall performance comparison covers 147 Easy conditions and 85 Hard conditions per model. Actual tool-call and user-interaction statistics include both successful and failed runs, while fact-request statistics use runs with complete records.

    For reference gaps, EtE_t uses closed-state tasks with valid action references, and EiE_i uses Easy tasks with complete fact-request records and information references. Task success is no longer a prerequisite for inclusion in these statistics. Figure 5 further separates successful and failed runs to analyze execution and information-request behavior within each outcome group.

  • Revision 2: Adding comparisons on identical condition sets to control differences in model coverage.

    To further remove differences in record coverage, we recomputed results using conditions for which all models have valid records and references. This supplementary comparison includes both successful and failed runs, without filtering for shared success.

    Shared comparison scopeBase tasksConditions
    EtE_t: closed-state Easy tasks1868
    EiE_i: Easy tasks with complete information records32100
    ModelMean EtE_tMean EiE_i
    Astra+0.12+0.11
    Kimi K3+0.13−0.49
    HY4+0.24−0.34
    GPT-5.5−0.24−0.57
    Qwen3.8 Max+0.00−0.66
    Seed 2.1 Pro+0.03−0.64
    Gemini 3.5 Flash+0.25−0.53
    GLM-5.3+0.22−0.48
    DeepSeek V4 Pro−0.53−0.54

Negative values indicate fewer recorded operations or information requests than the reference requires, which may accompany early termination or failure to acquire necessary information. In addition, the action-reference scope for EtE_t remains limited to the 18 closed-state base tasks above; its conclusions are not generalized to all 63 base tasks.

  • Revision 3: Organizing the analysis around four questions and interpreting efficiency alongside task outcomes.

    Analysis questionResultsMain findings
    How does imperfect information affect task completion?Table 3Relative to task-matched Full baselines, Easy total scores decrease by 5.3–24.5 percentage points, and grounded scores decrease by 5.9–40.4 percentage points. Furthermore, 22.7% of successful Easy outputs receive ungrounded credit.
    How does imperfect information affect execution efficiency?Figures 3–5Actual tool calls increase by 14.4%–71.5% across models on Easy, while mean fact requests increase from approximately 0.0 to 1.4. Combining Et/EiE_t/E_i with successful/failed outcome groups helps distinguish necessary information recovery, additional calls, and insufficient information requests.
    How do task difficulty and perturbation severity affect performance and efficiency?Table 3; Figures 6–7Capability limitations and action-budget ceilings on Hard affect the interpretation of perturbation effects. For example, Gemini 3.5 Flash reaches the 100-call limit in all 30 Hard Full conditions without fully completing any task. We also report score and call-count changes across perturbation conditions.
    What are the main failure causes under imperfect information?Figures 8 and 13; Boxes 1–2Easy failures predominantly involve incorrect results, while Hard failures mainly involve incorrect or incomplete content and unusable submissions. The case studies further illustrate two behaviors: making decisions without clarification and repeatedly requesting already available information.

W3: A more detailed analysis of correct outputs produced without explicitly acquiring the required information.

We thank the reviewer for this suggestion. We have expanded the analysis in the revised Section 5 by jointly examining task outcomes, clarification records, acquired facts, and grading results.

Among 635 successful Easy runs under imperfect conditions, 144 (22.7%) receive ungrounded credit. These outputs pass the outcome checks despite incomplete recorded information supporting the credited decisions. This indicates that outcome correctness alone can conceal reliance on unconfirmed assumptions.

To place this grounding gap in context, Figure 8 summarizes failures under imperfect conditions, while Figure 13 examines whether relevant facts were acquired in failed or not fully grounded outputs. The failure distribution is:

TrackFailure categoryCountShare of failures within track
EasyIncorrect result33284.9%
EasyMissing prerequisite steps266.6%
EasyNo usable final answer338.4%
HardIncorrect or incomplete content30362.7%
HardIncorrect or missing files6814.1%
HardNo usable final answer11223.2%

The table aggregates the nine reported model rows and excludes Full conditions. The denominators are 391 Easy failures and 483 Hard failures.

We further identify two recurring behaviors through the concrete examples in Boxes 1–2.

  • Making decisions without clarifying unresolved information.

    Box 1 shows three models making definitive decisions about unresolved inputs without asking the user:

    ModelConditionObserved behavior
    GPT-5.5AW02/C1Excludes Sam’s conditional contribution and orders four camping pads.
    AstraWB07/C1Excludes a pending metric revision and creates no task.
    HY4AW06/C1Charges payment card DISC-B without confirming the designated card.

    All three corresponding Full runs pass, whereas these imperfect-condition runs fail task-relevant checks. These examples expose how unresolved information is converted into concrete decisions. Together with the ungrounded successful outputs, they motivate checking the information supporting a decision alongside its eventual correctness.

  • Repeatedly requesting information already available.

    Box 2 shows the complementary behavior: models ask users to reconfirm facts already provided in the task files.

    ModelConditionObserved behavior
    AstraAW14/R2Asks whether a credit has been applied, although the task file states that it is effective.
    HY4WB16/M2Reconfirms meeting-duration and notification policies already supplied.
    Kimi K3AW04/FullAsks for the delivery address already confirmed in the task file.

    The Astra and Kimi K3 tasks pass but incur avoidable information requests. These cases show that successful completion can also coexist with inefficient information use.

    The expanded analysis therefore reveals two distinct problems.

    • Agents may act on unresolved information when clarification is needed.
    • Agents may request information unnecessarily when it is already available.

Thanks for the attentive reading of the manuscript and constructive feedback. We have incorporated these changes into our revised version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

Dear Reviewer,

As the author-reviewer cycle nears its conclusion, we noticed that we have not yet received your response to our rebuttal. We invested considerable effort in comprehensively addressing each of your concerns, including conducting additional experiments, clarifying technical details and methodological aspects of the paper, and revising the manuscript accordingly.

We would be deeply grateful if you could review our detailed rebuttal and consider whether the substantial improvements and clarifications we have provided address your initial concerns. Your feedback is invaluable to us, and we believe our responses demonstrate significant enhancements to the work's quality and rigor.

Given the approaching deadline, we respectfully request your reconsideration of the manuscript in light of these updates. We sincerely appreciate your time and expertise throughout this process.

Thank you for your review, and we always look forward to your response.

Reviewer_3ifbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces EA-Bench, a benchmark for how efficiently agents recover from imperfect user instructions. The authors adapt 18 synthetic task families (from JobBench, AppWorld-UL, and WorkBench scenarios) into dependency graphs, each with 15 variants (Full, four levels each of Missing / Redundant / Conflicting, two Mixed), for 270 conditions. Records, policies, and graders are fixed across variants. Two lower bounds make efficiency measurable: an action minimum (required graph operations + one batched question) and an information minimum (hidden facts, defined via "counterfactual policy worlds"). Agents clarify with a Qwen3-8B user simulator, and the authors separate "grounded" success (correct and all required facts obtained) from raw correctness. Across nine models, accuracy drops from 76.5% (Full) to 60.2% (Mixed), many correct Missing outputs are ungrounded, and matched successes use 23.6% more tool calls.

Critical strengths
  • [Section 5.2, 6.1] Grounded vs. ungrounded correctness is the most interesting contribution, in my opinion.
    • E.g., only 22 of Qwen's 41 correct Missing outputs actually obtained the needed info.
    • It shows outcome-only grading overstates how well agents handle underspecification, a point that applies beyond this benchmark.
    • Counterfactual policy worlds (Prop. 2) are a principled way to define necessary facts.
  • [Section 3.1, Appendix B] Well-controlled paired design.
    • Only the instruction varies; records, graph, and grader are fixed.
    • The authors are upfront that paired conditions aren't independent samples.
Critical weaknesses
  • [Section 4.1, Appendix A] The scale and realism are too limited for the general conclusions.
    • Variants share records and policy values, so effectively there are 18 distinct tasks.
    • The tasks are small: K = 4–9 actions (e.g., procurement buys 12 units for 120 CNY).
    • Single attempt per model per condition, with no repeats.
    • Nine-model matched comparisons rest on 20 (tools) / 17 (info) shared tasks, and some Table 3 cells have n = 2–3.
    • The delta over AppWorld-UL (which already varies user knowledge and measures interaction burden) and τ/τ²-bench needs a larger, more varied task set.
  • [Section 3.3, 5.1, Appendix F–G] The evaluation infrastructure is entangled with the headline numbers.
    • The Qwen3-8B simulator + annotator are unvalidated against humans (H.3 admits this), and the I.4 example shows the simulator volunteering unrequested facts.
    • "Scope errors ... remain evaluation-system failures": simulator errors count against models.
    • Runtime errors count as failures (32/270 for GPT-6-Astra, 24 for GPT-5.5), and about half recover on retry. This alone can reorder Table 2.
    • Tasks and dialogues are in Chinese (only stated in Appendix F.1). This matters for model comparisons and should be in the main text.
    • MiniMax-M2.5 at 20.4% (vs. 59–80% for others) looks like a harness/format issue rather than capability.
  • [Abstract, Section 6.2] The "847.5% more information items" headline is misleading.
    • It compares Full (0 required facts) to Mixed (up to 4 required). Mean excess requests actually decrease (Section 6.2).
    • I'd drop it from the abstract or replace it with the excess number.

[Overall] Good question, and the grounded/ungrounded distinction is worth publishing in some form. But 18 small synthetic families, single attempts, an unvalidated 8B simulator whose failures count against models, and a misleading headline don't support the paper's conclusions. These need a substantially larger benchmark and human validation, not a revision.

Questions for authors

Clarifications:

  1. [Section 3.3] How reliable is the simulator on a sample: how often does it refuse legitimate questions, leak unrequested facts, or give wrong values? Same for the annotator: how many "annotation errors" are the annotator's fault?
  2. [Table 2] How do rankings change if runtime errors are treated as missing and re-run, rather than counted as failures?
  3. [Appendix F.1] Any results on an English version? Would rankings hold?
  4. [Section 5.1] What explains MiniMax-M2.5's low accuracy? Is it the one-JSON-action protocol / "first report ends the episode, even if malformed"?
  5. [Section 4.2] Is asking one fact per turn (arguably more natural) penalized by the single-batched-question action minimum?

Suggestions:

  1. Multiple attempts with CIs, and varied records / policy values / wording within families (as Appendix B suggests).
  2. A small human-as-user study (cf. Seshadri et al., 2026, which the paper cites).

Minor:

  • [Table 1] The coverage marks didn't render clearly in my copy.
  • [Limitations] What are the "Codex-based runs"? The main text describes a custom JSON protocol.
  • [Appendix] Move internal artifacts (file paths, hashes, historical snapshots) to the code release.
  • [Table 2] The snapshot timestamp in the caption is unnecessary.
5 comments

We thank the reviewer for recognizing the value of distinguishing grounded from ungrounded correctness, our counterfactual definition of necessary facts, and the controlled paired design. We appreciate the opportunity to address the concerns raised and respond below.

W1 [Section 4.1, Appendix A] The scale and realism are too limited for the general conclusions.

We thank the reviewer for raising this concern. We agree that the scale and complexity of the original task set were insufficient to support broad general conclusions. During the rebuttal period, we therefore expanded and revised the benchmark in the following three aspects.

  • Improvement 1: Expanding the base-task set and strengthening validity and realism checks.

    We expanded the benchmark from 18 to 63 base-task instances and provide the complete DAG abstractions for all base tasks in Appendix C. The tasks are adapted from JobBench, AppWorld-UL, and WorkBench. While preserving the original task objectives and solvability, we construct imperfect-information variants through DAG-guided counterfactual analysis.

    We also introduce a two-stage checking process. First, AI-assisted checks assess whether missing facts remain inferable, whether conflicting statements have a clear authority ordering, and whether redundant information preserves the original facts and task requirements. These checks are combined with counterfactual and executable checks to verify variant validity. Human experts then screen the variants for realistic user communication scenarios. Ultimately, we retain 232 evaluation conditions from 792 candidates, comprising 63 Full baselines and 169 imperfect variants.

  • Improvement 2: Increasing task complexity and reporting results by task difficulty.

    The revised benchmark divides tasks into Easy and Hard tracks according to their inherent structure and deliverable requirements:

    TrackBase tasksWorkflow DAG nodesMissingRedundantConflictMixedTotal
    Easy334–1345144114147
    Hard3013–3010319585

EA-Bench's hard track's tasks involve databases, multi-file analysis, and complex artifact generation. For example, JB17 requires analyzing 3,930 ITSM ticket records, combining asset inventories and project budgets, and producing a four-sheet workbook and a gap-register CSV.

In our experiments, some models cannot fully complete any Hard task even under Full conditions. For example, Gemini 3.5 Flash reaches the budget limit of 100 tool calls in all 30 Hard Full conditions, yet fails to fully complete any of these tasks. This demonstrates that, even with complete information and a reasonably generous execution budget, the added Hard tasks pose a substantial challenge to current advanced models.

  • Improvement3: Clarifying the statistical units and scope of conclusions while broadening task coverage in the main comparisons.

We evaluate model performance separately on the Easy and Hard tracks. For each model, the overall Easy score covers 33 base tasks and 147 conditions, while the overall Hard score covers 30 base tasks and 85 conditions. This reduces the dependence of our main conclusions on a small set of shared successful tasks.

W2 [Section 3.3, 5.1, Appendix F–G] The evaluation infrastructure is entangled with the headline numbers.

We thank the reviewer for highlighting how evaluation infrastructure may affect model comparisons. We agree that the simulator’s knowledge boundaries, runtime error handling, and evaluation language should be clearly specified and distinguished from the models’ task performance. The updated results are summarized in Table below.

ModelEasy Total / GHard Total / G
Astra82.3 / 81.655.9 / 55.9
Kimi K377.6 / 63.313.2 / 12.4
HY476.9 / 68.010.8 / 10.5
GPT-5.568.0 / 54.415.0 / 14.2
Qwen3.8 Max65.3 / 51.712.2 / 11.4
Seed 2.1 Pro63.9 / 49.012.6 / 12.2
Gemini 3.5 Flash59.2 / 44.20.0 / 0.0
GLM-5.354.4 / 46.312.0 / 11.2
DeepSeek V4 Pro†46.3 / 37.412.6 / 11.8
Overall results. G denotes grounded score. Easy: task pass rate (%); Hard: mean rubric score (0–100).

We have made the following revisions.

  • Improvement 1: Upgrading the user simulator and grading model, and explicitly defining the simulator’s knowledge boundaries.

    In the revised version, we replace Qwen3-8B with the stronger GPT-5.5 (medium effort) for user simulation and artifact grading. The user simulator follows the knowledge-bounded design of AppWorld-UL. We provide it with predefined task-specific user knowledge and authoritative facts, and explicitly restrict it to answering questions relevant to the current query using this knowledge, without providing a complete solution.

    We also separately record the information items requested by the agent and the facts actually returned by the simulator. Only returned facts count as user information acquired by the agent; asking a question without receiving an answer does not count as information acquisition. This design is now explicitly described in Sections 3 and 4.3 of the main text.

  • Improvement 2: Fixing the identified runtime issues and updating the evaluation results.

    The runtime errors identified in the original version have been fixed, and the affected results have been updated following recovery. For service or response issues, we rerun the corresponding tasks after recovery. For artifact-grading issues, we preserve the original artifacts and regrade them. The revised results table uses the final outcomes obtained after this recovery process.

    We also distinguish infrastructure errors from model execution failures. For example, our error categorization distinguishes action-budget exhaustion, incorrect submissions, and grading-program errors.

  • Improvement 3: Standardizing the evaluation language to English and stating this explicitly in the main text.

    In the revised version, task instructions, input files, and user-facing tool messages are all in English, as are user interactions. This setting is now explicitly stated in Section 4.3, Models and Configuration, of the main text.

  • Improvement 4: Analyzing failure causes using execution, clarification, and grading records.

    The revised version presents outcome categories under imperfect conditions in Figure 8 and further examines failed or not fully grounded outputs in Figure 13. Easy failures predominantly involve incorrect results, whereas Hard failures mainly involve incorrect or incomplete content and unusable submissions. By combining clarification records with specific check outcomes, we further analyze two typical failure behaviors:

    • Failure mode 1: Making decisions without clarifying unresolved information. Box 1 presents examples from GPT-5.5, Astra, and HY4. All three models make definitive decisions about unresolved information without asking the user. Their corresponding Full runs pass, whereas the runs under imperfect conditions fail task-relevant checks.
    • Failure mode 2: Repeatedly requesting information already provided. Box 2 presents examples from Astra, HY4, and Kimi K3. Although the task files already contain the required facts, the models ask users to confirm them again, introducing avoidable information requests. Some of these tasks ultimately pass, but interaction efficiency is reduced.

    These results show that models may guess and act when clarification is needed, while also making unnecessary queries when sufficient information is already available.

[Abstract, Section 6.2] The "847.5% more information items" headline is misleading.

We thank the reviewer for pointing out that this statement may be misleading. We agree that Full and Mixed conditions impose different information requirements: an increase in actual requests may reflect the recovery of necessary information and cannot be directly interpreted as additional waste or reduced efficiency. We have therefore removed “information items increase by 847.5%” from the revised abstract and now separately analyze actual tool calls, requested facts, and efficiency metrics relative to task-specific references.

  • Revision 1: Definition of efficiency metrics

    The revised manuscript reports actual tool-call counts in Figure 3, requested fact counts in Figure 4, and EtE_t and EiE_i, separately for successful and failed runs, in Figure 5:

    Et=A−Aref,Ei=Q−Iref,E_t=A-A^{\mathrm{ref}},\qquad E_i=Q-I^{\mathrm{ref}},

    where AA and QQ denote the actual numbers of tool calls and requested facts, respectively, while ArefA^{\mathrm{ref}} and IrefI^{\mathrm{ref}} denote the action and information reference requirements for the corresponding task condition. This distinguishes additional information required by the task from information requested beyond the reference requirement.

  • Revision 2: Changes in absolute costs

    The actual costs on Easy are summarized below. Following the aggregation procedure in the main text, we first average the perturbation-column means within each model and then average equally across the nine models. These statistics include both successful and failed runs; fact-request counts use only runs with complete records.

    Actual cost metricFullImperfect conditions
    Mean tool calls10.7314.03
    Mean requested facts0.0281.393

    Tool-call counts increase by 14.4%–71.5% across models. For fact requests, we report changes in absolute counts rather than a potentially misleading percentage increase calculated from a near-zero Full baseline.

    Meanwhile, Figure 5 separates successful and failed runs within the eligible subset for each metric. The ranges of model-level mean gaps across the nine models are:

    MetricSuccessful runs: range of model meansFailed runs: range of model means
    EtE_t−0.07 to +0.22−1.26 to +0.65
    EiE_i−0.32 to +0.15−1.66 to −0.18

    Mean EiE_i is negative for failed runs across all nine models and is consistently lower than for the corresponding successful runs, indicating fewer requested facts than the reference requires. Therefore, fewer information requests or a lower gap does not automatically indicate greater efficiency; it may also accompany missing necessary information and task failure.

  • Revision 3: Conclusions from model evaluation

    The revised conclusion emphasizes that imperfect information reduces task completion and increases actual action costs by 14.4%–71.5% across models on Easy, while increasing the mean number of requested facts from approximately 0.0 to 1.4. These increases include necessary information-recovery costs introduced by the task conditions; whether they constitute excess overhead must be assessed using EtE_t and EiE_i.

    More concerning, 22.7% of successful Easy task completions receive ungrounded credit, indicating that models may complete tasks by guessing unconfirmed facts. We therefore jointly report task completion, grounding, and execution efficiency to provide a more complete evaluation of personal agents under imperfect information.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Overview

Dear PC, SAC, AC, and all reviewers,

We sincerely thank you for your thoughtful reviews and constructive feedback. We appreciate the reviewers’ recognition of our controlled paired design, the distinction between grounded and ungrounded correctness, and the joint evaluation of action and information efficiency.

We have prepared point-by-point responses, expanded the benchmark, and revised the experimental analysis around task completion, execution efficiency, task difficulty and perturbation severity, and failure behavior. The main additions and clarifications are summarized below.

AreaAdditions and revisionsLocationReviewer IDs
Task coverage and validationExpanded from 18 to 63 base-task instances, adding 15 application tasks and 30 professional workflows. Combined AI-assisted checks with expert screening to retain 232 conditions from 792 candidates, comprising 63 Full baselines and 169 imperfect variants.Section 4.1; Table 2; Appendix B3ifb, UH6z, Yc7K, dicD
Task complexity and DAG documentationIntroduced Easy and Hard tracks with 33 and 30 base tasks, respectively, and DAGs containing 4–13 and 13–30 nodes. Hard includes multi-file and artifact-generation tasks, such as analyzing 3,930 ITSM records. Provided complete DAGs and node descriptions for all 63 base tasks.Section 4.1; Table 2; Figure 2; Appendices B–C3ifb, pxL4, Yc7K, dicD
Simulator and evaluation languageReplaced Qwen3-8B with GPT-5.5 (medium effort) for user simulation and artifact grading. Specified predefined user-knowledge boundaries and separate records for requested and returned facts. Made the English-language evaluation explicit in the main text.Sections 3, 4.2, and 4.33ifb, UH6z, Yc7K, dicD
Infrastructure recovery and record completenessRecovered identified service/response failures and regraded affected preserved artifacts. Updated the result table and clarified that incomplete fact-request records do not automatically imply task failure; 40 such records retain successful task outcomes.Table 3; Appendix A.2; reviewer responses3ifb, Yc7K, dicD
Grounded and ungrounded performanceUpdated total and grounded results for both tracks. Found that 144 of 635 successful Easy runs under imperfect conditions (22.7%) receive ungrounded credit, highlighting the gap between outcome correctness and information-supported success.Section 5; Table 3; Figures 9–103ifb, pxL4, UH6z, Yc7K, dicD
Actual costs and headline reportingRemoved the 847.5% information-request headline. Separately reported tool calls, requested facts, and user interactions. The revised results show 14.4–71.5% more tool calls across models on Easy and mean fact requests increasing from approximately 0.0 to 1.4.Abstract; Sections 5–6; Figures 3–4 and 113ifb, UH6z, dicD
Successful and failed executionsIncluded failed runs in cost accounting and separated successful/failed reference gaps. Added a supplementary comparison without shared-success filtering using 68 common Easy conditions for EtE_t and 100 for EiE_i, covering 18 and 32 base tasks, respectively.Sections 4.2 and 5; Figures 5 and 12; reviewer responses for the common-condition comparison3ifb, pxL4, dicD
Reference scope and question batchingClarified the protocol-specific scope of action and information references, the distinction between requested facts and interaction turns, and the batching assumption. Retained actual costs for Hard tasks without assigning unvalidated global reference gaps.Sections 3 and 4.2; Figures 2, 4, and 11; reviewer responses3ifb, pxL4, UH6z, Yc7K
Difficulty and perturbation analysisReported task-matched score and tool-call changes across perturbation levels. Discussed task composition, performance floors, and action-budget ceilings when interpreting non-monotonic trends and Hard-task results.Section 5; Table 2; Figures 6–7; reviewer responses3ifb, pxL4, UH6z
Failure analysis and diagnostic examplesAdded outcome and information-acquisition analyses, with examples across models of acting on unresolved information without clarification and reconfirming facts already available.Section 5; Figures 8 and 13; Boxes 1–23ifb, pxL4, UH6z, Yc7K
Statistical uncertaintyReported 95% intervals from 5,000 source-cluster bootstrap resamples for paired performance differences.Appendix A.3; Table A13ifb, UH6z, dicD
Interpretation and reproducibilityClarified operational efficiency versus subjective user experience and the role of expert task screening. Improved comparison-table notation.Table 1; Section 4.3; reviewer responses; source package3ifb, UH6z, dicD

Thank you again for your time and consideration. We welcome further discussion of these additions and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.