- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a unified framework for evaluating LLM agents across heterogeneous benchmarks, UniACE, through a common "instruction-tool-environment" interface, a shared execution harness. To my understanding, UniACE aims to preserve native tasks and success criteria while standardizing efficiency measurement and failure attribution, with optional offline snapshots for reproducible evaluation.
Critical strengths
- This paper integrates seven benchmarks into a unified evaluation pipeline with standardized execution.
- Large-scale experiments reveal substantial configuration sensitivity.
Critical weaknesses
- Limited novelty. I do not agree that "a unified evalution setting" is a novel idea. 1) Some of the previous work mentioned in the paper is also fair. 2) Study such as "General Agent Evaluation", "AgentCompass" all supports these fair settings.
- Ranking reversals demonstrate configuration sensitivity, but do not establish that UniACE produces more reliable or representative comparisons. Different model may performs best in different harness. "fix the evaluation CONDITION" "ELIMINATE the impact of conditions".
- Changes to tool interfaces, evidence sources, and database engines may alter task difficulty. Task equivalence and individual component effects remain insufficiently validated.
Questions for authors
See Weaknesses.
We thank the reviewer's review. We address the weaknesses and questions below
W1:
UniACE’s contribution is the end-to-end standardization of benchmark specification, agent execution, and evaluation for reusable model comparisons. The distinction from existing integration platforms is discussed in §2, lines 137-144 and summarized in Appendix A/Table 4. General Agent Evaluation connects heterogeneous agents and benchmarks through protocol adapters while preserving their internal implementations (GAE, §2.2). AgentCompass similarly treats the harness as an independently configurable component and uses different harnesses across benchmarks in its experiments (AgentCompass, §3.2 and 4.1). A shared integration protocol does not itself standardize the internal agent execution procedure
UniACE integrates heterogeneous benchmarks through a common instruction–tool–environment specification and evaluates models using a shared, task-agnostic execution core (§3.1–3.3, lines 159-173, 179-198, and 201-231; Appendices B–D, lines 727-730, 790-803, 842-859, and 866-947). Its smolagents-based implementation standardizes model invocation, tool execution, error handling, and interaction management, together with isolated task runtimes and consistent success, efficiency, and failure records. Task-specific instructions, tools, and agent roles remain configurable within this shared protocol
This standardization supports both convenient benchmark integration and subsequent model reevaluation. New or updated benchmarks can follow the common specification and reuse the execution and evaluation pipeline without modifying the core framework. Registered models and evaluation settings can then be selected through configuration files, allowing the integrated benchmarks to be rerun under the same declared protocol (§3.1, lines 152-155 and 168-173; Appendix C, lines 842-859; Appendices I.1–I.2, lines 2839-2889 and 3190-3235). The integration of seven benchmarks demonstrates this design across single-agent, multi-agent, and safety evaluation. Our contribution therefore comprises the implemented standardization, the reusable evaluation resource, and the empirical analyses it enables (§4.2–4.6, lines 309-323, 348-355, 406-413, 424-434, and 445-461)
W2:
UniACE makes the execution conditions consistent across the evaluated models. All models use the same core agent workflow, with task instances, tools, environment settings, and evaluation criteria held fixed within each benchmark condition (§3.1–3.3, lines 168-173, 179-198, and 205-213). This provides an explicit, reusable basis for comparing models. As stated in §1, lines 73-74, and §3.2, lines 184-186, the framework controls these conditions rather than eliminating their effects
The ranking reversals illustrate why execution conditions matter; our contribution is to standardize those conditions for model comparison. The resulting benefit is that researchers can inspect the evaluation procedure and repeat comparisons using the same tasks, execution core, and resource settings. Our reported rankings describe performance under this shared UniACE protocol. They do not imply that the same ranking must hold under every other harness
W3:
UniACE preserves the original benchmarks’ core task design, interaction logic, and success requirements. All migrated components and evaluation procedures were manually checked and corrected against the originals. Appendix H, lines 2062-2071, 2426-2429, and 2597-2607 documents the benchmark-specific adaptations, including the task assets and functional requirements retained through tool-interface and database changes
Our experiments examine framework changes and resource changes through separate comparisons. §4.2/Table 1 compares results reported under the source benchmark frameworks with those obtained under UniACE. Appendix E.1/Table 6 further holds the task, model, retrieval backend, and search-tool implementations fixed to compare framework execution behavior, with trajectories showing differences in prompt construction, memory representation, error feedback, and termination. §4.3/Table 2 compares online and offline resource settings within the same UniACE framework, reporting both task performance and resource usage
UniACE supports both live online retrieval and fixed offline evaluation. Appendix E.2, lines 1117-1129 and 1141-1144 documents inaccessible reference pages, outdated content, stale snippets, and changing retrieval results. Offline snapshots preserve the relevant evidence and fix its representation across runs, providing repeatable, inspectable resource conditions (§3.2, lines 188-193). §4.3/Table 2 reports the performance and resource-use differences associated with online and offline evidence settings. Researchers can therefore retain live-web evaluation or reuse fixed snapshots for reproducible model comparisons