acceptodds
Under review as a conference paper at ICLR 2027

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Abstract

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.

Primary area:
datasets and benchmarks
Keywords:
General Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pmpSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniaBench, a benchmark for evaluating general AI agents across diverse application scenarios. It constructs 1431 tasks (with a 644-task challenging subset) spanning 90 level-1 and 354 level-2 domains across consumer, business, and employee settings. Tasks are synthesized via four routes (DAG, DAG-S, Solver, Program) and evaluated using rubric-based scoring and programmatic verification. The paper evaluates 24 closed- and open-source models, reporting that even frontier models achieve only approximately 58% Pass@1 on the challenging set. Extensive robustness analyses validate the evaluation protocol across judge models and user simulators.

Critical strengths
  1. The paper proposes a diverse benchmark for General AI Agents, offering broader domain coverage and higher authenticity compared to previous benchmarks.
  2. The construction methodology is advanced. The multi-path task construction (including DAG, DAG-S, Solver, and Program) effectively evaluates diverse agent capabilities, such as multi-turn interaction, optimization, and code execution.
  3. Regarding experimental design, the authors conducted a comprehensive evaluation of 24 models. The fine-grained diagnostic breakdown by capability, domain, and efficiency demonstrates the benchmark's difficulty and discriminative power.
Critical weaknesses
  1. Appendix A.1 mentions that the consistency with human evaluators exceeds 99%. Please provide more detailed information on the evaluation methodology, including the sample size, the number of evaluators, and the specific evaluation protocol.
Questions for authors
  1. The paper does not mention the cost of reproduction and evaluation. Could the authors clarify whether the overall work is easy to reproduce?
4 comments
AuthorsOpenReview ↗

Part 1/2 Human evaluation and token cost

We are grateful for your detailed review and constructive feedback. We will respond to your concerns below.

W1: Human evaluation methodology, sample scope, and review procedures

Appendix A.1 reports human–judge agreement, while Appendix C.1 describes the human review process. We provide further details below.

Approximately 4,000 initial candidate tasks underwent execution validation, human review, and quality filtering, producing the full collection of 1,431 tasks. We subsequently selected 644 tasks for the challenging set. This final selection preserves fine-grained coverage while controlling evaluation scale and is separate from filtering invalid or ambiguous candidates.

Fourteen reviewers received unified instructions and task-specific training. Before formal review, a pilot annotation and calibration stage covered task formulation, environment support, rubric design, capability labels, and difficulty assessment. Reviewers worked in seven pairs; every retained task was reviewed by at least two people, with disagreements referred for adjudication.

StepProcedure
1. Task and environmentCheck request clarity, information sufficiency, and whether environment states and tools support the required operations.
2. SolvabilityExecute the golden action sequence in the actual task environment; require all rubric items to pass or the corresponding verifier to accept.
3. Rubric reviewDouble-check coverage, validity, and objectivity; identify missing requirements and remove unnecessary or subjective checkpoints.
4. Trajectory inspectionCover all tasks, inspecting trajectories from at least five distinct evaluated models per task.
5. Human–judge comparisonAssess each rubric requirement using the same task evidence and compare human and automated binary decisions.
6. AdjudicationReview and correct identified issues, revalidate candidates, and filter out problems that cannot be reliably resolved.

Task and rubric review assesses dataset quality, while inspection of execution results checks automated judgments. The candidate and retained-task counts above describe dataset curation; the agreement statistic uses rubric-item decisions on manually inspected trajectories as its unit.

Specifically, agreement above 99% is the number of valid inspected rubric items with matching human and automated pass/fail decisions divided by the total number of valid inspected items. Since task success requires every item to pass, this verification checks the decisions underlying the strict success criterion. The five-judge experiment in Appendix A.1 provides complementary cross-judge evidence.

Q1: Evaluation cost and ease of reproduction

Table 2 reports ToolSteps and UserTurns, and Figure 8 further analyzes execution efficiency. For cost planning, the following table provides budget-derived token and API-cost figures for higher-cost frontier models and the three highest-ranked open-weight models.

ModelTokens per taskAPI cost per task
Claude Sonnet 58.63M$31.06
GPT-5.6 Sol6.90M$49.69
GLM-5.27.67M$15.35
DeepSeek-V4-Pro7.81M$14.43
Kimi-K2.66.82M$10.63

M denotes one million tokens. Token counts include input and output, with thinking tokens included in output. Costs cover the evaluated agent, excluding user-simulator and rubric-judge calls and local infrastructure.

Our work is designed to be straightforward to reproduce through the provided evaluation infrastructure. The supplementary package includes all 644 challenging-set tasks, executable environments, rubrics, programmatic verifiers, model runners, four-route orchestration, and scoring utilities. Researchers can use these artifacts directly.

The README provides dependency installation, model-interface configuration, and execution commands. Environments run locally without access to live business applications or production accounts; API-based evaluation does not require deploying the evaluated models locally. The framework supports parallel execution, checkpointing and resume, task/route selection, and re-scoring saved trajectories without rerunning the agent.

The Reproducibility Statement on page 10 identifies the relevant materials: Sections 3.3 and 4.1 describe construction, metrics, model settings, and execution limits, with route-specific protocols and robustness analyses in the appendix. Researchers can first validate their configuration on a few tasks before running the complete challenging set.

AuthorsOpenReview ↗

Part 2/2 Additional details on the human evaluation

W1: Additional details on the human evaluation sample and agreement

We supplement the human evaluation description with explicit counts and the calculation underlying the reported agreement.

The construction process began with approximately 4,000 candidate tasks. Execution validation and human quality control produced the retained collection of 1,431 tasks, from which 644 were selected for the challenging set. These counts describe dataset curation and subset selection; the human–judge agreement statistic is calculated over annotated rubric-item decisions.

Human evaluation componentCount
Trained reviewers14
DAG tasks covered by trajectory inspection1,102
Evaluated models represented per taskAt least 5
Trajectories inspectedMore than 5,510
Valid annotated rubric-item decisions used for agreement9,808
Human–judge matching decisions9,727
Human–judge disagreements81
Item-level agreement99.17%

The broader inspection of trajectories and the annotated units used to calculate agreement are separate counts. Each agreement unit is one binary rubric decision on one inspected trajectory; the same requirement assessed on different trajectories contributes separate units.

Specifically:

Human–judge agreement = 9,727 / 9,808 × 100% = 99.1741% ≈ 99.17%.

Reviewers received unified instructions and task-specific training, followed by pilot annotation and calibration. They were organized into seven pairs for double review of retained tasks and rubrics, with disagreements referred for adjudication. Review covered task clarity, environment support, execution of golden action sequences, and the coverage and objectivity of rubric requirements.

For the agreement assessment, human evaluators used the same task evidence and rubric criteria as the automated judge to determine whether each requirement was satisfied. This directly checks the binary decisions underlying the strict all-rubric-items-pass criterion.

Appendix C.1 describes the review organization and procedures, while Appendix A.1 reports human agreement and complementary five-judge robustness analyses. The counts above provide the sample scope and explicit numerator and denominator for the human agreement result.

We hope our responses clarify your concerns. Looking forward to any further feedback.

Thanks to the authors for their thorough and well-structured rebuttal.

Regarding W1, the additional quantitative breakdown—specifically the 9,727 matching decisions out of 9,808 valid annotated rubric items, the coverage of all 1,102 DAG tasks with at least five model trajectories each, and the clear six-step review protocol—provides the methodological transparency I requested. Clarifying that the >99% figure refers to item-level binary pass/fail agreement rather than task-level success is helpful and resolves my concern.

Regarding Q1, the budget-derived token and API cost table directly answers my question about evaluation expenses. The description of the released artifacts, including executable environments, rubrics, verifiers, and orchestration utilities, alongside the note that environments run locally without live production accounts, confirms that reproduction is feasible and well-supported.

The responses fully address my original concerns without introducing new issues. I have no further questions requiring clarification at this stage.

So I recommend maintaining my rating of 3 and confidence of 3. The authors successfully resolved my specific methodological and practical inquiries regarding human evaluation and reproduction costs. However, as noted in my original review, the presentation still requires substantial effort in places. Since the rebuttal addressed my targeted weaknesses and questions satisfactorily, I uphold my initial positive but cautious assessment.

Thank you for taking the time to read our rebuttal and for letting us know that it addressed your questions. We also appreciate your feedback on the presentation and will keep it in mind as we revise the paper.

Reviewer_NbhmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for general agents that covers a wide range of everyday app and work scenarios. Environments and tasks are generated with an LLM, checked by humans, and scored with rubrics or code checks. The authors test many models and find that even the best ones solve only about half the tasks

Critical strengths

The coverage is broad. The mix of domains, stateful environments, and single and multi-turn tasks goes beyond most existing agent benchmarks. The authors also check how stable the results are across judges and user simulators, which makes the leaderboard more believable

Critical weaknesses

Scoring relies on LLM judges, and a task counts as solved only if every rubric item passes, so judge noise can matter. The main judge and user simulator is also a model on the leaderboard. It would help to show that the model ranking holds across judges and to say more about how the human check was done. It would also help to know how many generated tasks turned out unsolvable or ambiguous, so readers can tell task problems from real agent failures

Questions for authors
  1. Does the model ranking stay the same under the different judges you tried?
  2. Can you share more detail on the human review, such as how many tasks were checked and how many were dropped or fixed?
5 comments

We are grateful for your detailed review and constructive feedback. We will respond to your concerns below.

W1: Evaluation reliability and task validity

Evaluation reliability was a central consideration in designing OmniaBench. We distinguish two auxiliary models: the user simulator participates in interactions and may affect information disclosure and execution trajectories, whereas the rubric judge assesses completed trajectories against fixed criteria.

Section 4.3 and Figure 8(c) report the user-simulator ablation. Replacing the user model preserves the relative ranking of the three evaluated agents, with Kendall's tau of 1.0 and an average coefficient of variation of 7.24% in model scores. Appendix A.1, Tables 3–4, reports robustness analyses using five rubric judges on identical trajectories, covering both total scores and binary item decisions. Alternative judges and human verification also help examine whether evaluation results depend on the default judge, which is itself an evaluated model.

Appendix C.1 describes the human review process: 14 trained reviewers were organized into seven pairs, with each retained task reviewed by at least two people and disagreements referred for adjudication. Review covered task clarity, environment support, and rubric coverage, validity, and objectivity. We also inspected execution results from at least five distinct evaluated models per task; rubric-item-level human–judge agreement exceeded 99%.

For solvability, we require an executable golden action sequence: the reference tool-call sequence must complete the task in its actual executable environment and satisfy every rubric item, or pass the corresponding verifier for programmatically evaluated tasks. This complements the execution validation and human quality checks described in Section 3.3 and Figure 4.

A plausible reference answer alone is therefore insufficient. A candidate must pass execution validation and double human review of the task and its rubric items. Candidates that fail execution validation or have unresolved ambiguity or scoring issues are filtered out of the evaluation set.

Q1: Ranking stability across auxiliary models

We examined user simulators and rubric judges separately. The reported experiments support stable relative comparisons under the evaluated conditions.

For user simulators, Section 4.3 and Figure 8(c) directly compare rankings: the three evaluated agents retain the same relative ordering across three user models, with Kendall's tau of 1.0. Absolute scores vary moderately, while their ordering remains unchanged in this experiment.

For rubric judges, Appendix A.1 evaluates five models on the same trajectories and rubrics, reporting trajectory-level and item-level agreement. Holding trajectories fixed separates judge variation from changes in user interactions or agent execution.

Here, the judge assesses explicit task-completion conditions rather than open-ended preferences. Double human review checks rubric objectivity, and manual verification of actual judgments yielded over 99% rubric-item-level human–judge agreement. These checks provide human evidence for the strict all-items-pass criterion and complement the five-judge experiment.

Q2: Human review scope and procedures

We generated approximately 4,000 initial candidate tasks. Execution validation, human review, and quality filtering produced the full collection of 1,431 tasks, from which 644 were selected for the challenging set. This final selection controls evaluation scale while preserving fine-grained coverage; it is distinct from filtering out invalid or ambiguous tasks.

As described in Appendix C.1, 14 reviewers received unified instructions and task-specific training. A pilot annotation and calibration stage covered task formulation, environment support, rubric design, capability labels, and difficulty assessment. Reviewers worked in seven pairs; every retained task was reviewed by at least two people, with disagreements referred for adjudication.

  1. Task and environment. Reviewers checked whether requests were clear, necessary information was available, and environment states and tools supported the required operations.

  2. Solvability. The golden action sequence had to execute successfully in the actual task environment and satisfy all rubric items or pass the programmatic verifier.

  3. Rubric quality. Reviewers checked whether each item corresponded to a genuine task requirement, whether any requirements were missing, and whether checkpoints were unnecessary, subjective, or inconsistent with the available environment evidence.

  4. Capability and difficulty annotations. Reviewers checked that annotations matched task content rather than adopting model-generated labels without verification.

  5. Execution results and automated judgments. Human inspection covered trajectories from at least five distinct evaluated models per task, including their execution outcomes and automated judgments.

Identified issues had to be corrected and revalidated before a candidate could be retained. Candidates with ambiguity, inconsistency, or execution problems that could not be reliably resolved were filtered out. Tasks entering the evaluation set therefore had to pass both execution validation and human review.

AuthorsOpenReview ↗

Part 3/3 Additional stability results on the 644-task challenging set

Q1: Additional stability results on the 644-task challenging set

We provide the following numerical breakdown using the official 644-task challenging subset. User models are substituted on its 354 DAG tasks, while outcomes on the other 290 single-turn tasks are held fixed.

Evaluated agentGPT-4.1 userDeepSeek-V4-Flash userDeepSeek-V4-Pro userSD (percentage points)
Qwen3.7-Max49.69%52.64%49.69%1.39
Qwen3.5-35B-A3B27.80%32.61%27.95%2.23
Qwen3-235B-A22B-Thinking19.41%22.67%20.03%1.41

The three agents retain the same relative ranking under every user model, with pairwise Kendall's tau of 1.0. The default-user scores match the paper's leaderboard. Figure 8(c) reports the full-set analysis over 1,102 DAG tasks, whereas the table above reports overall scores on the 644-task challenging set. The mean coefficient of variation is 5.72% for this table, compared with 7.24% for the full-set DAG analysis.

For rubric judges, Appendix A.1 separately evaluates five models on fixed trajectories and rubrics. It reports total-score ICC(A,1) of 0.780 and mean pairwise Spearman correlation of 0.821, alongside binary rubric-item agreement statistics. These results concern judge substitution; the ranking table above concerns user-simulator substitution.

Q2: Additional counts for human verification

To make the scope of human checking explicit, the following counts distinguish dataset curation, trajectory inspection, and the item-level agreement calculation.

ComponentScope
Initial candidate poolApproximately 4,000 tasks
Retained collection / challenging subset1,431 / 644 tasks
Trajectory-level human inspection1,102 DAG tasks; at least five evaluated models per task; more than 5,510 trajectories
Human reviewers14 trained reviewers, organized into seven pairs
Valid human-verified rubric-item decisions used for agreement9,808
Matching human and automated decisions9,727
Disagreements81

Human–judge agreement is therefore 9,727 / 9,808 × 100% = 99.1741%, rounded to 99.17%. This is item-level binary agreement, not agreement in aggregate scores. The 9,808 annotated decisions used for this calculation are distinguished from the broader trajectory-inspection coverage above.

As described in Appendix C.1, task and rubric review uses double checking and adjudication. Retained tasks must have executable reference solutions, and unresolved ambiguity, inconsistent criteria, or execution problems lead to filtering. The reduction from approximately 4,000 candidates to 1,431 retained tasks reflects the overall curation process; selection of the 644-task challenging set is a separate stage.

We hope our responses clarify your concerns. Looking forward to any further feedback.

Reviewer_S5jPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniaBench evaluates general-purpose agents across consumer, business, and workplace scenarios. It introduces a taxonomy covering 354 fine-grained domains, executable stateful environments, and four task-construction routes spanning multi-turn interaction, optimization, and procedural execution. The dataset contains 1,431 tasks, with 644 selected for the main evaluation. Contributions also include ten capability dimensions and eight difficulty factors. Experiments compare multiple models, assess execution efficiency and failure patterns, and examine sensitivity to user simulators, judges, and repeated runs.

Critical strengths
  1. Substantial benchmark resource. The combination of broad domain coverage, executable environments, and four complementary task-construction routes provides a useful resource for studying heterogeneous agent workflows. The supplementary package includes all 644 evaluation tasks and an evaluation harness, enabling independent inspection.

  2. The paper examines efficiency, repeated-run reliability, user-simulator sensitivity, and judge agreement. These analyses provide useful context for interpreting agent performance and strengthen the benchmark’s empirical contribution.

Critical weaknesses
  1. The scoring protocol and its reliability need stronger verification. The paper requires all rubric items to pass, but compute_scores.py uses a threshold of 0.6, whereas route_scores.py requires a full score. This discrepancy does not establish that the published leaderboard is incorrect, but leaves the reproduction pathway unclear. Furthermore, judge agreement on total scores does not establish stability of the all-items-pass metric.

  2. The diagnostic claims exceed the supporting evidence. Capability scores aggregate task success across overlapping labels, making specific ability deficits difficult to distinguish from task composition and difficulty. The appendix acknowledges this limitation, but the main discussion still draws relatively strong conclusions about capability boundaries. Sparse domain coverage—one DAG task per second-level domain in the challenging set—and missing task-level capability annotations in the released package further limit validation of these conclusions.

Questions for authors
  1. Which scoring script and output fields produced the reported leaderboard and domain scores? Please reconcile the 0.6 versus full-score thresholds and provide the exact reproduction commands.

  2. How do task-level Pass@1 and model rankings change when the judge is replaced? For the reported >99% human agreement, what were the sample size, sampling procedure, and unit of agreement?

  3. Can you provide the task-level capability annotations and exact aggregation procedure used for the capability profiles? What evidence supports attributing differences to specific capabilities rather than differences in the associated task sets?

8 comments

We are grateful for your detailed review and constructive feedback. We will respond to your concerns below.

W1: Scoring protocol and reliability of strict Pass@1

The reported leaderboard uses the all-rubric-items-pass criterion: a task is successful only when every rubric item is satisfied. The final evaluation pipeline invokes route_scores.py, which requires a full score, consistent with the pass criterion in result_evaluator.py. The 0.6 threshold was used for exploratory analysis, not for the reported leaderboard, as explained in Q1.

To assess the reliability of strict Pass@1, we conducted the five-judge robustness experiment and manually checked the underlying judgments. All rubrics underwent double human review to check coverage of task requirements, remove unnecessary checkpoints, identify omissions, and avoid subjective criteria. Human inspection of execution results covered all tasks, with trajectories from at least five distinct evaluated models examined for each task. The reported human–judge agreement exceeds 99% at the rubric-item level. Since task success is determined by these binary item decisions, this human verification provides evidence supporting the reliability of the strict success criterion.

W2: Capability profiles, domain coverage, and task-level annotations

Capability labels were incorporated during task construction and reviewed by humans. Each capability dimension contains a substantial proportion of single-label tasks primarily targeting that capability. Multi-label tasks capture joint capability requirements in realistic workflows: a deficiency in any required capability may prevent successful completion. These labels describe task requirements; they do not attribute a failure simultaneously to every associated capability.

As stated in Appendix B.1, a capability profile represents performance on tasks involving the corresponding capability, rather than a causal measurement of an isolated ability.

The selection of one DAG task per second-level domain should be understood in relation to our taxonomy's granularity. These categories are sufficiently fine-grained to approach specific task types. The challenging set is selected from the full collection to cover every such category, while domain-level analyses aggregate multiple tasks at the first-level domain. These two levels serve fine-grained coverage and broader domain comparison, respectively.

We retain the task-level annotations internally. Our open-source preparation already includes each task's labels and the Python scripts used for aggregation, and these data and scripts will be included in the open-source release.

Q1: Scoring entry point and the purpose of the 0.6 threshold

The final evaluation uses pass_at_1 from route_scores.py. For rubric-based tasks, it reads rubric_eval.avg_result; for programmatically verified tasks, it reads verifier_eval.score. Both require a full score to count as a pass. Domain scores aggregate the same binary task-success criterion, consistent with result_evaluator.py.

The 0.6 threshold in compute_scores.py was used to study partial completion by smaller models. Because these models rarely completed entire long-horizon tasks, we explored different thresholds to examine their performance at different completion levels. This exploratory setting was not used to compute the final leaderboard. The formal scoring entry point and this exploratory analysis utility therefore serve different purposes.

Q2: User simulators, rubric judges, and human agreement

User simulators and rubric judges play different roles: the former participate in interactions and can affect execution trajectories, whereas the latter assess trajectories that have already been generated.

For user simulators, the ablation in Figure 8(c) shows that replacing the user model preserves the relative ranking of the three evaluated agents, with Kendall's tau of 1.0 and an average coefficient of variation of 7.24% in model scores.

For rubric judges, we evaluated the same trajectories using five models and manually reviewed both the rubrics and the resulting judgments. Fourteen trained reviewers participated in the human review process. Tasks and rubrics underwent double review, and human inspection of execution results covered all tasks, with trajectories from at least five distinct evaluated models examined for each task.

The reported agreement above 99% is rubric-item-level agreement: the proportion of valid inspected items for which the human and automated judge made the same binary pass/fail decision. Human verification and cross-judge experiments provide complementary evidence for evaluation reliability.

Q3: Capability annotations, aggregation, and interpretation

We can provide the task-level annotations used for the capability profiles. Our open-source preparation already includes each task's labels and the Python scripts used to compute the aggregate results, and all these data and scripts will be included in the open-source release.

The following example illustrates the annotation format:

Task IDTask excerptCapability labels (illustrative)
route1 / 784Check linked vehicles, the default parking vehicle, and verification status without making changes.Tool Use; Information Gathering

Capability requirements were deliberately incorporated during task construction and reviewed by humans. For example, tasks labeled with coding capability explicitly require the agent to write and execute code, such as running code through Bash. The label follows the task's designed requirements, rather than being assigned retrospectively because a particular model happened to use code. Apart from the corresponding coding requirements, the remaining data-construction procedures are consistent, including adjustable difficulty parameters.

Our profiles represent performance on tasks grouped by capability requirements. Single-label tasks primarily target the corresponding capability, while multi-label tasks capture requirements involving several capabilities, where a deficiency in any required capability may affect completion. Each task contributes to every capability dimension with which it is associated. The labels describe required capabilities rather than identify a unique capability responsible for an individual failure.

This task-requirement-based organization is similar in spirit to the skill categories in WorkArena++ and the combination of abilities required by GAIA.

AuthorsOpenReview ↗

Part 3/3 Additional results on evaluation and robustness

W1 & Q2: Additional results on evaluation stability and human verification

We provide additional results using the official 644-task challenging subset and clarify the human verification scope.

1. User-simulator substitution on the 644-task subset

User simulation applies to the 354 DAG tasks. We therefore substitute user models on these tasks while holding the other 290 single-turn task outcomes fixed. The default DeepSeek-V4-Pro results match the original records task by task, and the resulting overall scores match the paper's leaderboard (the figure is the full set result).

Evaluated agentGPT-4.1 userDeepSeek-V4-Flash userDeepSeek-V4-Pro user
Qwen3.7-Max49.69% (1st)52.64% (1st)49.69% (1st)
Qwen3.5-35B27.80% (2nd)32.61% (2nd)27.95% (2nd)
Qwen3-235B-Thinking19.41% (3rd)22.67% (3rd)20.03% (3rd)

The relative ranking is unchanged, with pairwise Kendall's tau of 1.0. The following table reports signed differences from the default DeepSeek-V4-Pro user and the population standard deviation across the three user models. All values are in percentage points.

Evaluated agentGPT-4.1 differenceDeepSeek-V4-Flash differenceStandard deviation
Qwen3.7-Max+0.00+2.951.39
Qwen3.5-35B-0.16+4.662.23
Qwen3-235B-Thinking-0.62+2.641.41

The mean coefficient of variation is 5.72% under this 644-task aggregation. The 7.24% reported in Figure 8(c) corresponds to the original 1,102-DAG-task analysis.

2. Rubric-judge robustness

Appendix A.1 evaluates the same trajectories and rubrics using five judges: DeepSeek-V4-Flash, GPT-4.1, DeepSeek-V4-Pro, GLM-5-Thinking, and Claude-Opus-4.7-Thinking. The reported results are:

LevelUnitsJudgesICC(A,1)Mean SpearmanMin SpearmanNMPDP95 NMPD
Trajectory total score1,10250.7800.8210.7560.0620.266
Weighted rubric-item score9,80850.7410.8560.8110.0070.060

For binary decisions on the 9,808 rubric items, all five judges agree on 86.3% of items; 9.3% exhibit a 4-vs-1 disagreement and 4.4% a 3-vs-2 disagreement. Fleiss' kappa is 0.573.

These statistics describe cross-judge agreement in total scores and individual rubric decisions. The user-simulator ranking results above and the rubric-judge results concern two distinct experimental settings.

3. Human verification scope and agreement

Human verification covered 1,102 tasks and trajectories from at least five distinct evaluated models, totaling more than 5,510 inspected trajectories. Fourteen trained reviewers participated, with double review and adjudication of disagreements.

Review covered task clarity, environment support, executable golden action sequences, and the coverage, validity, and objectivity of rubric requirements. Human reviewers also checked actual execution evidence against the rubric and compared their binary decisions with the automated judgments.

The unit of agreement is one rubric-item decision on one inspected trajectory. The same rubric applied to trajectories from different models therefore contributes separate verification units. Programmatic verifier outcomes are separate from the LLM rubric agreement calculation.

Rubric-item-level human–judge agreement is approximately 99.17%, calculated as:

Agreement = A / N × 100% ≈ 99.17%,

where N is the total number of valid manually verified rubric-item decisions and A is the number with matching human and automated pass/fail decisions. Equivalently, A ≈ 0.9917 × N. The denominator counts verified item decisions, rather than tasks or rubric definitions alone.

This verification directly examines the binary decisions underlying the all-rubric-items-pass success criterion and complements the cross-judge analyses in Appendix A.1.

We hope our responses clarify your concerns. Looking forward to any further feedback.

Comments on the paper

1 comment

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.