- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OmniaBench, a benchmark for evaluating general AI agents across diverse application scenarios. It constructs 1431 tasks (with a 644-task challenging subset) spanning 90 level-1 and 354 level-2 domains across consumer, business, and employee settings. Tasks are synthesized via four routes (DAG, DAG-S, Solver, Program) and evaluated using rubric-based scoring and programmatic verification. The paper evaluates 24 closed- and open-source models, reporting that even frontier models achieve only approximately 58% Pass@1 on the challenging set. Extensive robustness analyses validate the evaluation protocol across judge models and user simulators.
Critical strengths
- The paper proposes a diverse benchmark for General AI Agents, offering broader domain coverage and higher authenticity compared to previous benchmarks.
- The construction methodology is advanced. The multi-path task construction (including DAG, DAG-S, Solver, and Program) effectively evaluates diverse agent capabilities, such as multi-turn interaction, optimization, and code execution.
- Regarding experimental design, the authors conducted a comprehensive evaluation of 24 models. The fine-grained diagnostic breakdown by capability, domain, and efficiency demonstrates the benchmark's difficulty and discriminative power.
Critical weaknesses
- Appendix A.1 mentions that the consistency with human evaluators exceeds 99%. Please provide more detailed information on the evaluation methodology, including the sample size, the number of evaluators, and the specific evaluation protocol.
Questions for authors
- The paper does not mention the cost of reproduction and evaluation. Could the authors clarify whether the overall work is easy to reproduce?
4 comments
Part 2/2 Additional details on the human evaluation
W1: Additional details on the human evaluation sample and agreement
We supplement the human evaluation description with explicit counts and the calculation underlying the reported agreement.
The construction process began with approximately 4,000 candidate tasks. Execution validation and human quality control produced the retained collection of 1,431 tasks, from which 644 were selected for the challenging set. These counts describe dataset curation and subset selection; the human–judge agreement statistic is calculated over annotated rubric-item decisions.
| Human evaluation component | Count |
|---|---|
| Trained reviewers | 14 |
| DAG tasks covered by trajectory inspection | 1,102 |
| Evaluated models represented per task | At least 5 |
| Trajectories inspected | More than 5,510 |
| Valid annotated rubric-item decisions used for agreement | 9,808 |
| Human–judge matching decisions | 9,727 |
| Human–judge disagreements | 81 |
| Item-level agreement | 99.17% |
The broader inspection of trajectories and the annotated units used to calculate agreement are separate counts. Each agreement unit is one binary rubric decision on one inspected trajectory; the same requirement assessed on different trajectories contributes separate units.
Specifically:
Human–judge agreement = 9,727 / 9,808 × 100% = 99.1741% ≈ 99.17%.
Reviewers received unified instructions and task-specific training, followed by pilot annotation and calibration. They were organized into seven pairs for double review of retained tasks and rubrics, with disagreements referred for adjudication. Review covered task clarity, environment support, execution of golden action sequences, and the coverage and objectivity of rubric requirements.
For the agreement assessment, human evaluators used the same task evidence and rubric criteria as the automated judge to determine whether each requirement was satisfied. This directly checks the binary decisions underlying the strict all-rubric-items-pass criterion.
Appendix C.1 describes the review organization and procedures, while Appendix A.1 reports human agreement and complementary five-judge robustness analyses. The counts above provide the sample scope and explicit numerator and denominator for the human agreement result.
We hope our responses clarify your concerns. Looking forward to any further feedback.
Thanks to the authors for their thorough and well-structured rebuttal.
Regarding W1, the additional quantitative breakdown—specifically the 9,727 matching decisions out of 9,808 valid annotated rubric items, the coverage of all 1,102 DAG tasks with at least five model trajectories each, and the clear six-step review protocol—provides the methodological transparency I requested. Clarifying that the >99% figure refers to item-level binary pass/fail agreement rather than task-level success is helpful and resolves my concern.
Regarding Q1, the budget-derived token and API cost table directly answers my question about evaluation expenses. The description of the released artifacts, including executable environments, rubrics, verifiers, and orchestration utilities, alongside the note that environments run locally without live production accounts, confirms that reproduction is feasible and well-supported.
The responses fully address my original concerns without introducing new issues. I have no further questions requiring clarification at this stage.
So I recommend maintaining my rating of 3 and confidence of 3. The authors successfully resolved my specific methodological and practical inquiries regarding human evaluation and reproduction costs. However, as noted in my original review, the presentation still requires substantial effort in places. Since the rebuttal addressed my targeted weaknesses and questions satisfactorily, I uphold my initial positive but cautious assessment.
Thank you for taking the time to read our rebuttal and for letting us know that it addressed your questions. We also appreciate your feedback on the presentation and will keep it in mind as we revise the paper.
Part 1/2 Human evaluation and token cost
We are grateful for your detailed review and constructive feedback. We will respond to your concerns below.
W1: Human evaluation methodology, sample scope, and review procedures
Appendix A.1 reports human–judge agreement, while Appendix C.1 describes the human review process. We provide further details below.
Approximately 4,000 initial candidate tasks underwent execution validation, human review, and quality filtering, producing the full collection of 1,431 tasks. We subsequently selected 644 tasks for the challenging set. This final selection preserves fine-grained coverage while controlling evaluation scale and is separate from filtering invalid or ambiguous candidates.
Fourteen reviewers received unified instructions and task-specific training. Before formal review, a pilot annotation and calibration stage covered task formulation, environment support, rubric design, capability labels, and difficulty assessment. Reviewers worked in seven pairs; every retained task was reviewed by at least two people, with disagreements referred for adjudication.
Task and rubric review assesses dataset quality, while inspection of execution results checks automated judgments. The candidate and retained-task counts above describe dataset curation; the agreement statistic uses rubric-item decisions on manually inspected trajectories as its unit.
Specifically, agreement above 99% is the number of valid inspected rubric items with matching human and automated pass/fail decisions divided by the total number of valid inspected items. Since task success requires every item to pass, this verification checks the decisions underlying the strict success criterion. The five-judge experiment in Appendix A.1 provides complementary cross-judge evidence.
Q1: Evaluation cost and ease of reproduction
Table 2 reports ToolSteps and UserTurns, and Figure 8 further analyzes execution efficiency. For cost planning, the following table provides budget-derived token and API-cost figures for higher-cost frontier models and the three highest-ranked open-weight models.
M denotes one million tokens. Token counts include input and output, with thinking tokens included in output. Costs cover the evaluated agent, excluding user-simulator and rubric-judge calls and local infrastructure.
Our work is designed to be straightforward to reproduce through the provided evaluation infrastructure. The supplementary package includes all 644 challenging-set tasks, executable environments, rubrics, programmatic verifiers, model runners, four-route orchestration, and scoring utilities. Researchers can use these artifacts directly.
The README provides dependency installation, model-interface configuration, and execution commands. Environments run locally without access to live business applications or production accounts; API-based evaluation does not require deploying the evaluated models locally. The framework supports parallel execution, checkpointing and resume, task/route selection, and re-scoring saved trajectories without rerunning the agent.
The Reproducibility Statement on page 10 identifies the relevant materials: Sections 3.3 and 4.1 describe construction, metrics, model settings, and execution limits, with route-specific protocols and robustness analyses in the appendix. Researchers can first validate their configuration on a few tasks before running the complete challenging set.