acceptodds
Under review as a conference paper at ICLR 2027

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

Abstract

Agent benchmarks are increasingly used to compare large language models (LLMs) across domains, yet a reported score reflects a complete model–harness–environment configuration rather than the model alone. Benchmark packages couple native tasks with specific prompts, tool protocols, orchestration logic, and sometimes dynamic external resources, making cross-benchmark comparisons sensitive to implementation and resource conditions. We present UniACE, a unified framework for model-centric evaluation under an explicit, common execution condition. UniACE represents each benchmark as an instruction–tool–environment triplet, executes LLMs through a shared, task-agnostic harness in isolated per-task runtimes, and preserves native success criteria. For tasks that rely on dynamic resources, an optional offline mode replaces live access with fixed, pre-collected snapshots. Its evaluation protocol further standardizes efficiency measurement, execution records, and trace-based failure attribution. We migrate 7 benchmarks spanning 24 domains and evaluate 15 models in more than 400K rollouts consuming 5B tokens. Comparisons with source implementations show large bidirectional score changes and model-ranking reversals, while matched online and offline runs reveal substantial sensitivity to accessible evidence and its representation. Under the shared UniACE configuration, efficiency and failure profiles expose task-dependent model behaviors hidden by task-success scores alone. These findings motivate reporting agent benchmark outcomes as properties of an explicit evaluation configuration, enabling more interpretable and reproducible cross-benchmark comparisons.

Primary area:
datasets and benchmarks
Keywords:
LLM-based Agent, Evaluation, Framework

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_JMnyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a unified framework for evaluating LLM agents across heterogeneous benchmarks, UniACE, through a common "instruction-tool-environment" interface, a shared execution harness. To my understanding, UniACE aims to preserve native tasks and success criteria while standardizing efficiency measurement and failure attribution, with optional offline snapshots for reproducible evaluation.

Critical strengths
  1. This paper integrates seven benchmarks into a unified evaluation pipeline with standardized execution.
  2. Large-scale experiments reveal substantial configuration sensitivity.
Critical weaknesses
  1. Limited novelty. I do not agree that "a unified evalution setting" is a novel idea. 1) Some of the previous work mentioned in the paper is also fair. 2) Study such as "General Agent Evaluation", "AgentCompass" all supports these fair settings.
  2. Ranking reversals demonstrate configuration sensitivity, but do not establish that UniACE produces more reliable or representative comparisons. Different model may performs best in different harness. "fix the evaluation CONDITION" ≠\ne "ELIMINATE the impact of conditions".
  3. Changes to tool interfaces, evidence sources, and database engines may alter task difficulty. Task equivalence and individual component effects remain insufficiently validated.
Questions for authors

See Weaknesses.

1 comment

We thank the reviewer's review. We address the weaknesses and questions below


W1:

UniACE’s contribution is the end-to-end standardization of benchmark specification, agent execution, and evaluation for reusable model comparisons. The distinction from existing integration platforms is discussed in §2, lines 137-144 and summarized in Appendix A/Table 4. General Agent Evaluation connects heterogeneous agents and benchmarks through protocol adapters while preserving their internal implementations (GAE, §2.2). AgentCompass similarly treats the harness as an independently configurable component and uses different harnesses across benchmarks in its experiments (AgentCompass, §3.2 and 4.1). A shared integration protocol does not itself standardize the internal agent execution procedure

UniACE integrates heterogeneous benchmarks through a common instruction–tool–environment specification and evaluates models using a shared, task-agnostic execution core (§3.1–3.3, lines 159-173, 179-198, and 201-231; Appendices B–D, lines 727-730, 790-803, 842-859, and 866-947). Its smolagents-based implementation standardizes model invocation, tool execution, error handling, and interaction management, together with isolated task runtimes and consistent success, efficiency, and failure records. Task-specific instructions, tools, and agent roles remain configurable within this shared protocol

This standardization supports both convenient benchmark integration and subsequent model reevaluation. New or updated benchmarks can follow the common specification and reuse the execution and evaluation pipeline without modifying the core framework. Registered models and evaluation settings can then be selected through configuration files, allowing the integrated benchmarks to be rerun under the same declared protocol (§3.1, lines 152-155 and 168-173; Appendix C, lines 842-859; Appendices I.1–I.2, lines 2839-2889 and 3190-3235). The integration of seven benchmarks demonstrates this design across single-agent, multi-agent, and safety evaluation. Our contribution therefore comprises the implemented standardization, the reusable evaluation resource, and the empirical analyses it enables (§4.2–4.6, lines 309-323, 348-355, 406-413, 424-434, and 445-461)


W2:

UniACE makes the execution conditions consistent across the evaluated models. All models use the same core agent workflow, with task instances, tools, environment settings, and evaluation criteria held fixed within each benchmark condition (§3.1–3.3, lines 168-173, 179-198, and 205-213). This provides an explicit, reusable basis for comparing models. As stated in §1, lines 73-74, and §3.2, lines 184-186, the framework controls these conditions rather than eliminating their effects

The ranking reversals illustrate why execution conditions matter; our contribution is to standardize those conditions for model comparison. The resulting benefit is that researchers can inspect the evaluation procedure and repeat comparisons using the same tasks, execution core, and resource settings. Our reported rankings describe performance under this shared UniACE protocol. They do not imply that the same ranking must hold under every other harness


W3:

UniACE preserves the original benchmarks’ core task design, interaction logic, and success requirements. All migrated components and evaluation procedures were manually checked and corrected against the originals. Appendix H, lines 2062-2071, 2426-2429, and 2597-2607 documents the benchmark-specific adaptations, including the task assets and functional requirements retained through tool-interface and database changes

Our experiments examine framework changes and resource changes through separate comparisons. §4.2/Table 1 compares results reported under the source benchmark frameworks with those obtained under UniACE. Appendix E.1/Table 6 further holds the task, model, retrieval backend, and search-tool implementations fixed to compare framework execution behavior, with trajectories showing differences in prompt construction, memory representation, error feedback, and termination. §4.3/Table 2 compares online and offline resource settings within the same UniACE framework, reporting both task performance and resource usage

UniACE supports both live online retrieval and fixed offline evaluation. Appendix E.2, lines 1117-1129 and 1141-1144 documents inaccessible reference pages, outdated content, stale snippets, and changing retrieval results. Offline snapshots preserve the relevant evidence and fix its representation across runs, providing repeatable, inspectable resource conditions (§3.2, lines 188-193). §4.3/Table 2 reports the performance and resource-use differences associated with online and offline evidence settings. Researchers can therefore retain live-web evaluation or reuse fixed snapshots for reproducible model comparisons

Reviewer_TtCEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UniACE, a unified framework for evaluating LLM agents under a shared execution protocol. It standardizes benchmark representation, provides isolated per-task runtimes, supports offline snapshots for dynamic resources, and collects task-completion, efficiency, and failure-analysis metrics. Across seven benchmarks covering 24 domains, 15 models, and over 400K rollouts, the authors demonstrate substantial score changes and model-ranking reversals under different evaluation configurations. The paper highlights the importance of explicitly specifying evaluation conditions when comparing agent capabilities.

Critical strengths
  1. Broad and practically useful evaluation infrastructure. UniACE integrates seven established benchmarks and systematically records task success, resource usage, and execution traces. The large-scale evaluation provides useful empirical evidence about the limitations of comparing agent scores across different execution configurations.

  2. Meaningful analysis of evaluation sensitivity. The reported score shifts, ranking reversals, and online–offline differences illustrate that agent performance depends on both the execution framework and accessible environmental evidence, motivating more transparent and reproducible evaluation practices.

Critical weaknesses
  1. Comparisons across evaluation settings are insufficient to explain the causes of score changes. The paper reports differences in model scores and rankings between the original implementations and UniACE, but these settings may differ in execution workflows, tools, and environments simultaneously. The current experiments do not clearly disentangle the effects of these factors, limiting the conclusions that can be drawn about the impact of the unified framework itself.

  2. The validity of evaluation results after benchmark migration remains insufficiently established. UniACE integrates diverse agent benchmarks into a common framework, but these benchmarks differ in task execution and environmental conditions. In particular, online and offline evaluations may provide agents with different information. Further validation is needed to establish whether the migrated tasks preserve the evaluation objectives and difficulty of the original benchmarks.

Questions for authors
  1. Could the authors conduct controlled experiments to isolate the effects of execution frameworks and environments on evaluation results, thereby better explaining changes in model scores and rankings?

  2. Could the authors systematically compare the original benchmarks with UniACE on the same models and tasks, reporting score agreement and model ranking correlation to validate the reliability of the migrated evaluations?

1 comment

We thank the reviewer for recognizing the practical value of the unified framework. We address the weaknesses and questions below.


W1 and Q1:

The submission includes complementary comparisons of framework-level and resource-level changes. §4.2/Table 1 compares the source benchmark configurations with UniACE. As stated in §4.2 (lines 255-260), these aggregate gaps are interpreted as differences between evaluation configurations, rather than attributed to an individual harness component.

Appendix E.1/Table 6 provides a more closely matched framework comparison. On BFCL Web Search task #61, the task, Gemini-2.5-Flash model, retrieval backend, and search-tool implementations are held fixed. Under the same twenty-step budget, the original framework terminates unsuccessfully after three steps, whereas UniACE reaches the reference answer after five or eight steps, depending on the retrieval-output variant. The accompanying trajectories document how the two frameworks construct context, handle errors, and determine termination under otherwise matched task, model, and retrieval settings.

§4.3/Table 2 separately compares online and offline execution within the shared UniACE framework. Holding the framework fixed allows us to examine sensitivity to the resource condition. As explicitly discussed in §4.3 (lines 348-355), offline snapshots change accessible evidence and its representation, so the observed differences are not attributed solely to environmental stability. Together, these comparisons distinguish the framework-level mechanisms illustrated in Appendix E.1 (lines 985-991 and 1002-1008) from the resource-level differences examined in Table 2.


W2 and Q2:

UniACE preserves the original benchmarks’ core task design, interaction logic, and success requirements. All migrated components and evaluation procedures were manually checked and corrected against the originals. Appendix H, lines 2062-2071, 2426-2429, and 2597-2607 documents the benchmark-specific adaptations, including the task assets and functional requirements retained through tool-interface and database changes.

UniACE supports both live online retrieval and fixed offline evaluation. Appendix E.2, lines 1117-1129 and 1141-1144 documents inaccessible reference pages, outdated content, stale snippets, and changing retrieval results. Offline snapshots preserve relevant evidence and fix its representation across runs, providing repeatable, inspectable resource conditions (§3.2, lines 188-193). Table 2 explicitly reports the performance and resource-use differences between online and offline evidence settings. Researchers can therefore retain live-web evaluation or select fixed resources for reproducible comparisons.

§4.2/Table 1 complements this validation with a systematic comparison of the source benchmark results and UniACE for the corresponding reference models. The “Orig.” columns contain officially reported scores, while the “Ours” columns report evaluations through UniACE’s shared execution protocol. These side-by-side results show the score differences and model orderings under the two configurations. The manual checks establish consistency in the retained benchmark requirements, while Table 1 examines the performance changes associated with executing those tasks through the unified framework.

Reviewer_FBeFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds UniACE, a tool for testing LLM agents in one common setup. Each benchmark is turned into three parts: instructions, tools, and environment. All models run with the same agent code, each task runs in its own separate space, and web tasks can use saved offline copies of web pages. The paper reports success rate, cost (steps, tokens, time), and types of failure. The authors test 15 models on 7 benchmarks with over 400K runs. They find that scores go up or down compared with the original benchmarks, that model rankings can flip, that offline web tasks score much higher than online ones, and that models with similar scores can differ a lot in cost.

Critical strengths
  1. The paper shows clearly that an agent score depends on the whole setup, not just the model. The ranking flips and the BFCL #61 case study make this point well.
  2. Moving seven very different benchmarks into one system is a lot of work. The configs, judge prompts, and offline data could be useful to others.
Critical weaknesses
  1. The paper does not show that its setup is more “model-centric.” One fixed agent setup is still just one setup, and it may help some models more than others. The paper only shows that changing the setup changes the scores, which past work (e.g., HAL, General Agent Evaluation) already showed. To support the main claim, the authors should test more than one shared setup.

  2. The paper does not check that the moved benchmarks still match the originals. Many parts change at once: some tools are rewritten or written by an LLM, Postgres is replaced by SQLite, offline web pages are summarized by an LLM, and some rule-based checkers seem to be replaced by LLM judges. So it is hard to know why scores differ from the original. The higher offline scores may come from cleaner, easier evidence, not from a more stable environment.

Questions for authors
  1. For each benchmark, which tasks use the original rule-based checker and which use an LLM judge? Where both exist (e.g., BFCL, τ-bench), how often do they agree?
  2. Did you rerun the “Orig.” scores with the same model versions, or copy them from the original papers?
  3. Do the ranking results still hold with a second, different shared agent setup?
  4. How many runs did you do for each setting, and how much do the results vary?
2 comments

We thank the reviewer for recognizing the practical value of the unified framework. We address the weaknesses and questions below.


W1 and Q3:

By “model-centric,” we mean comparing models through a common agent execution protocol shared across heterogeneous benchmarks (§3.1–§3.3, lines 159-173, 179-198, and 205-213). Within each benchmark condition, task instances, tools, environment settings, and evaluation criteria are held fixed while the evaluated model changes. Across benchmarks, task-specific instructions, tools, and agent roles may differ, but the core execution procedure remains shared. This distinction is explicitly stated in §3.2 (lines 179-186).

HAL and General Agent Evaluation have established that agent scaffolds affect model performance. These findings motivate our standardization effort; our contribution is the implemented protocol and cross-benchmark integration. UniACE combines an instruction–tool–environment specification, isolated runtimes, optional fixed-resource conditions, and consistent success, efficiency, and failure records (§3, lines 159-173, 176-193, and 201-231; Appendices B–D, lines 727-730, 790-803, 842-859, and 866-947). Our implementation builds on smolagents, an open-source, general-purpose, model-agnostic framework, and applies the same core protocol across the evaluated models without selecting or optimizing a separate scaffold for each backbone.

Our experiments explicitly examine model-dependent differences across execution configurations. On τ-bench Retail, GPT-4o changes from 61.2% under the source configuration to 28.7% under UniACE, while GPT-3.5-Turbo changes from 20.0% to 55.7%, reversing their ordering (§4.2; Table 1). Appendix E.1/Table 6 further examines execution behavior while holding the task, model, retrieval backend, and search tools fixed. Alongside the resource-use and failure analyses (§4.4–4.5, lines 406-413 and 424-434), these results characterize the implications of the declared evaluation configurations. The methodological claim is that models are evaluated through the same declared execution core across benchmarks. Cross-harness ranking stability is not the criterion by which we define “model-centric”; our comparisons are explicitly conditioned on the shared protocol (§3.2, lines 184-186).


W2:

UniACE preserves the core task design, interaction logic, and success requirements of the original benchmarks. All migrated components and evaluation procedures were manually checked and corrected against the originals to ensure consistency. These adaptations make the benchmarks executable through the shared interface while retaining their underlying functional logic. The benchmark-specific migration procedures are documented in Appendix H, lines 2062-2071, 2426-2429, and 2597-2607.

The offline mode is designed to make evaluation reproducible. Appendix E.2, lines 1117-1129 and 1141-1144 documents problems with live resources, including inaccessible reference pages, outdated content, stale search snippets, and changing retrieval results. These can prevent agents from accessing the evidence required by the benchmark’s fixed reference answers and undermine reproducibility. Offline snapshots preserve the relevant resources and fix their representation across runs, providing stable, auditable conditions for repeated evaluation (§3.2, lines 188-193).

UniACE supports both live online retrieval and fixed offline evaluation. While §4.2/Table 1 compares the original benchmark frameworks with UniACE, §4.3/Table 2 separately reports the original benchmark setting alongside UniACE’s online and offline results, showing differences in task performance and resource usage. We interpret the online–offline score differences as reflecting changes in evidence access and representation, rather than attributing them solely to environmental stability (§4.3, lines 348-355). Researchers can therefore retain live-web evaluation or select fixed snapshots when reproducible comparisons are required.

Q1:

UniACE distinguishes output-level evaluation from deterministic environment-state verification (§3.3, lines 205-213; Appendix D.1, lines 866-886). For state-supervised tasks, the reference trajectory is replayed from the same initial state, and the resulting terminal state is checked by hash matching (Eqs. 3–4). Where both output and state requirements apply, the output score is multiplied by the state-validity indicator (Eq. 5); a favorable LLM judgment alone is insufficient.

The migrated output-level checks mainly use LLM judges, with their actual prompts provided in Appendix I.3. These cover AgentBench Database, Knowledge Graph, Lateral Thinking Puzzle, and Web Browsing (lines 3311-3637); BFCL function-calling, memory, multi-turn, web-search, and hallucination tasks (lines 3640-4594); τ-family service tasks (lines 4597-4666); BrowseComp (lines 4668-4689); and MultiAgentBench scenarios (lines 4692-5033). Preserving benchmark-defined success requirements therefore does not mean retaining every upstream checker implementation unchanged. MultiAgentBench’s inherited rubrics and score normalization are described in Appendix H.5.1, lines 2440-2462. Agent-SafetyBench retains the original ShieldAgent evaluator and evaluation prompt (Appendix H.6.4, lines 2758-2768).

Recent studies provide quantitative evidence for human-aligned LLM evaluation. Arena-Hard reports 98.6% correlation with human preference rankings [1]. Trust or Escalate achieves over 80% agreement with human judgments at 79.1% coverage on ChatArena [2], while RULERS reports a quadratic weighted κ of 0.7122±0.0056 with GPT-4o on ASAP 2.0 [3]. These results establish precedents for validating LLM-based evaluation against human judgments.

Our validation uses the original benchmarks’ reference labels and success requirements to assess rollout outcomes (Appendix F/Table 7). We sampled 300 rollouts stratified across benchmarks and agents. Two annotators independently labeled each sample using these references, with disagreements resolved by a third. Gemini-3-Flash achieved Cohen’s κ=0.95 against the adjudicated human labels, compared with 0.93 for GPT-5-mini, 0.91 for Qwen3-235B-A22B, and human inter-annotator κ=0.97. This high agreement demonstrates consistency with human assessments based on the original benchmark references and supports using Gemini-3-Flash for the specified output-level evaluations.

References

[1] T. Li et al., “From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline,” in Proceedings of the 42nd International Conference on Machine Learning, vol. 267, 2025, pp. 34209–34231.

[2] J. Jung, F. Brahman, and Y. Choi, “Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement,” in Proceedings of the International Conference on Learning Representations, 2025.

[3] Y. Hong, H. Yao, B. Shen, W. Xu, H. Wei, and Y. Dong, “From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges,” in Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 2026, pp. 13001–13021.


Q2:

The “Orig.” scores in Table 1 are taken directly from the benchmarks’ official leaderboards or original papers; we did not rerun them using the same model versions (§4.2; Table 1). We retain these reported scores as reference baselines under their original evaluation frameworks to compare the results obtained under the source implementations with those obtained under UniACE’s unified execution protocol.


Q4:

Our evaluation across 15 models and seven benchmarks already consumed approximately 5B tokens. Repeating the full suite was prohibitively expensive within our available computational budget, so each model–benchmark–condition configuration was evaluated once. We set temperature to 0 (§4.1, lines 249-251) to reduce sampling randomness. Similar cost constraints are documented by HAL [4], which relies on single runs for most evaluations, and General Agent Evaluation [5], which runs each configuration once.

[4] S. Kapoor et al., “Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation,” in Proceedings of the International Conference on Learning Representations, 2026.

[5] E. Bandel et al., “General Agent Evaluation,” arXiv preprint arXiv:2602.22953, 2026.

Reviewer_ab2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced a framework to evaluate llm agentic capabilities. The framework is called UniACE. It represents each benchmark as instruction-tool-env triplet then executes LLMs through a harness and preserve native success criteria. It also standardizes efficiency metrics, execution records and failure attribution. The authors migrate 7 benchmarks across 24 domains and evaluated 15 models. It's a unified work for comparing models efficiently and effectively.

Critical strengths
  1. Benchmarking different models on different evaluation sets are quite painful, given there're many nuances across different benchmarks. The authors proposed this unified framework, making it easier evaluate and compare different models
  2. The case studies are convincing.
  3. The engineering is substantial. It migrates seven benchmarks like multi-agent, safety benchmarks.
Critical weaknesses
  1. Novelty is limited. This paper is mainly combining different benchmarks into a unified one.
  2. Re Experimental results, where are the Orig. numbers from? Is it from source papers or re-run by authors?
  3. Some benchmarks are outdated and useless to track.
  4. How to make sure it's easy to maintain?
Questions for authors

see weaknesses

1 comment

We thank the reviewer for recognizing the practical value of the unified framework. We address the weaknesses and questions below.


W1:

UniACE’s contribution goes beyond benchmark aggregation through the execution protocol and evaluation resource it provides. Related frameworks, including General Agent Evaluation and AgentCompass, support comparisons involving different models and scaffolds. UniACE implements a shared execution core across seven heterogeneous benchmarks, integrating an instruction–tool–environment representation (§3.1, lines 157-166; Appendix B, lines 727-730 and 751-755), isolated per-task runtimes (§3.2, lines 188-198), optional fixed-resource evaluation, and consistent task-completion, efficiency, and failure records (§3.3, lines 201-231; Appendix D, lines 866-891, 895-917, and 936-947). Its contribution lies in implementing and studying this combination across the included task families, with model comparisons explicitly conditioned on the chosen execution protocol.

The empirical contribution also extends beyond score comparisons. Appendix E.1/Table 6 compares two frameworks while holding the task, model, live retrieval backend, and tools fixed, revealing different termination behavior. §4.4–4.5 (lines 406-413 and 424-434) further expose resource-use differences and failure profiles hidden by task-completion scores alone. These analyses illustrate the diagnostic value of the implemented protocol beyond benchmark aggregation.


W2:

The “Orig.” scores in Table 1 are taken directly from the benchmarks’ official leaderboards or original papers; we did not rerun them using the same model versions (§4.2; Table 1). We retain these reported scores as reference baselines under their original evaluation frameworks to compare the results obtained under the source implementations with those obtained under UniACE’s unified execution protocol.


W3:

The selected benchmarks and their subsequent versions continue to be used in evaluations of leading models and recent agent research. For example, Qwen3.5 reports results on BFCL-V4, τ²-bench, and BrowseComp [1]; Claude Sonnet 4.6 evaluates τ²-bench and BrowseComp [2]; and GPT-5.6 also uses BrowseComp to assess agentic browsing [3]. Their adoption extends beyond model releases: MultiAgentBench is used in Explicit Trait Inference for Multi-Agent Coordination [4], while Agent-SafetyBench is used to evaluate agentic safety training in AgentDoG 1.5 [5]. AgentBench-derived environments also remain in use through AgentBench-fc in AgentRL [6]. This continued adoption supports their relevance across tool use, retrieval, multi-agent coordination, and safety. Our selection covers these complementary evaluation dimensions (§4.1, lines 238-243), while the framework supports integrating additional benchmarks (§3.1, lines 152-155; §4.6, lines 445-453).

[1] Qwen Team. Qwen3.5-397B-A17B Model Card. 2026.

[2] Anthropic. Claude Sonnet 4.6 System Card. 2026.

[3] OpenAI. GPT-5.6: Frontier Intelligence That Scales with Your Ambition. 2026.

[4] Abdurahman et al. Explicit Trait Inference for Multi-Agent Coordination. 2026.

[5] Liu et al. AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security. 2026.

[6] Zhang et al. AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework. 2025.


W4:

UniACE simplifies maintenance through its standardized instruction–tool–environment interface (§3.1, lines 157-166; Appendix B, lines 727-730 and 751-755). To integrate or update a benchmark, users only need to adapt its instructions, tools, and environment resources to this specification, without modifying the core framework code. The resulting benchmark can directly reuse the shared execution and evaluation pipeline. Model serving, resource paths, and execution settings are specified through configuration files that facilitate evaluations with various models and different specific evaluation settings (Appendix C, lines 842-859; examples in Appendices I.1–I.2, lines 2839-2889 and 3190-3235). Benchmark updates therefore require adapting the benchmark specification rather than revising the underlying framework. The integration of seven heterogeneous benchmarks demonstrates the applicability of this design across different task types.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.