acceptodds
Under review as a conference paper at ICLR 2027

OL-Bench: A Comprehensive Benchmark for Evaluating AI Agents in Omni-Life Scenarios

Abstract

Large language model (LLM)-based agents are increasingly applied to real-life interactive scenarios, where they are required to understand evolving user needs, follow environmental constraints, interact over multiple turns, and use tools to complete tasks. However, existing benchmarks often provide limited scenario coverage, simplify interactive environments, and primarily evaluate final task completion, creating a significant gap between benchmark evaluation and real-life application settings. Consequently, they may fail to faithfully reflect agents' actual performance in practice. In this work, we present Omni-Life Benchmark (OL-Bench), a comprehensive interactive benchmark for evaluating agents across diverse real-life scenarios. OL-Bench adopts an Entity–Tool–Instruction hierarchy to synthesize executable interactive environments. Based on these environments, dependency-aware multi-turn tasks are constructed with explicitly planned answers and multi-level distractors across four difficulty levels, ranging from easy to extreme. These tasks are executed through a scenario-specific Harness that coordinates environment state transitions, tool use, and interaction trajectories. OL-Bench contains 1055 tasks across 218 real-life scenarios, with an average of 46.4 tools and 20.2 Skills per scenario. It further provides 18 metrics across seven capability dimensions to evaluate both task completion and interaction processes. Our comprehensive evaluation shows that GPT-5.6-Sol, the best-performing model, achieves a Full Task Completion Rate of only 18.0%, underscoring the difficulty of complex real-life interactions for current agents. These results demonstrate the value of OL-Bench for systematically evaluating agent capabilities and diagnosing their limitations in real-life interactive scenarios.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Interactive Benchmark, Multi-turn Interaction, Tool Use

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fSHUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OL-Bench, an interactive benchmark for agents in diverse, real-life scenarios. This benchmark synthesizes environments from an Entity-Tool-Instruction hierarchy. Environments have scenario-specific skills, tool perturbations introduce variations in tools, and dependency-aware multi-turn tasks across different difficulty levels. Additionally, they introduce an interaction evaluation framework with 18 metrics across 7 dimensions. The authors evaluate several models and show that their benchmark presents a challenge for current agents.

Critical strengths

The authors present a pipeline for generating realistic environments and tasks to evaluate agents in diverse and realistic scenarios. When evaluating they present interesting analyses, about how different models have different strengths/weaknesses and why models fail.

Critical weaknesses

This paper presents a pipeline for environment and task synthesis, and demonstrates that models achieve an 18% task completion rate. The authors randomly sample 40 environments and 100 tasks for manual inspection, and correct any issues they identify. Beyond that, there doesn’t seem to be much analysis of whether the tasks and environments they synthesize are valid or truly solvable.

One claim of the paper is that the benchmark presented has diverse and realistic scenarios. However, there is no analysis that proves this claim. The authors can consider presenting metrics/analyses that compare the diversity and realism of their benchmark with existing benchmarks.

Questions for authors

n/a

5 comments
AuthorsOpenReview ↗

Response to Reviewer fSHU (1/3)

Dear reviewer, thank you very much for your time and effort in reviewing our paper. We try to address your concerns as follows and hope we can ease your concerns.


Q1: How are the validity and solvability of LLM-synthesized environments and tasks ensured?

A1: OL-Bench ensures the reliability of synthesized environments and tasks through deterministic execution checks, answer-grounded task construction, and iterative human validation, rather than relying solely on LLM judgments.

(1) Environment Validity. Each synthesized environment undergoes deterministic execution tests to verify tool functionality, entity consistency, and environment-state updates. Environments failing these checks are rejected.

(2) Task Solvability. We first define the answer requirements for each subtask and then generate entities satisfying these requirements, ensuring the existence of valid answers by construction. Automated checks further verify data consistency, entity relationships, and tool support, rejecting tasks that fail validation.

(3) Iterative Human Quality Validation. We adopt a multi-round generation → inspection → error analysis → framework refinement → regeneration → revalidation process. In each round, identified issues are addressed at the synthesis-framework level, followed by random resampling from regenerated environments and tasks, rather than merely rechecking previously identified errors. Across this process, five annotators inspect 40 environments and 100 tasks, with each sample independently reviewed by three annotators, achieving 91.2% agreement (Fleiss' κ = 0.82). We identify and correct 23 errors, incorporating the improvements into subsequent environment and task generation.

(4) Full-Scale Human Audit. We conduct a comprehensive human audit of the generated candidate environments and tasks, ultimately retaining 218 environments and 1,055 tasks after quality screening. Six annotators are divided into two independent groups of three, with each group reviewing all candidates without intergroup communication. The audit covers entity and data consistency, tool correctness, task executability, and Checklist validity. The two groups achieve 97.8% agreement (Cohen's κ = 0.94). For samples with conflicting assessments, all six annotators jointly review the cases, identify potential issues, and correct or discard invalid environments and tasks.

Together, these mechanisms provide complementary validation of environment executability, task feasibility, and synthesized artifact quality.

We have added the detailed validation procedures and human audit results to Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fSHU (2/3)

Q2: How does OL-Bench establish real-world representativeness and validate the realism of its synthesized environments?

A2: (1) Real-World Scenario Coverage. OL-Bench is grounded in real-world service platforms, public documentation, and operational experience from a deployed industrial agent for local-life services. Statically, real user requests inform task categories, difficulty levels, and query styles. Dynamically, industrial agent practices guide the design of Skills, Harness, and tool execution mechanisms. Based on these references, OL-Bench covers 12 service categories, 218 environments, and 1,055 tasks, spanning representative services from Meituan, Dianping, Taobao, JD.com, Ctrip, and 12306 [1–4], as detailed below.

CategoryRepresentative PlatformsEnvsTasks
TransportationCtrip, 1230630134
HealthcareMeituan, JD.com20132
AccommodationCtrip, Meituan1689
Leisure & EntertainmentMeituan, Dianping2188
RetailTaobao, JD.com2082
Daily-Life ServicesMeituan1571
Beauty & WellnessMeituan, Dianping1566
Pet ServicesMeituan1457
Food & DeliveryMeituan1154
FitnessMeituan629
Education & FamilyMeituan429
Composite ScenariosCross-Domain Services46224
Total12 Categories2181,055

Compared with LocalSearchBench, VitaBench, and TRIP-Bench [5-7], OL-Bench provides complementary coverage of pet services, beauty and wellness, fitness, education, and cross-domain interactions, including 46 composite environments and 224 tasks.

(2) Environment Realism. The complexity of real-world service systems primarily involves four aspects: (i) Task Dependencies, where operations follow interdependent workflows; (ii) Entity Complexity, requiring multi-constraint filtering among interconnected candidates; (iii) Business Rules, governing operational requirements and prerequisites; and (iv) State Transitions, where actions modify environment states and affect subsequent operations.

As described in Section 3 (Benchmark Construction) of our paper, OL-Bench models these complexities through the Entity–Tool–Instruction hierarchy, which defines interconnected entities, executable tools, and operational rules. Information Graphs capture task dependencies, L1/L2/L3 distractors introduce constraint-based selection challenges, and the Agent Harness manages tool execution and state updates.

(3) Human Validation. As detailed in Q1, we conduct iterative human validation on 40 environments and 100 tasks, achieving 91.2% agreement (Fleiss' κ = 0.82). We further perform a full-scale human audit using two independent groups of three annotators, achieving 97.8% agreement (Cohen's κ = 0.94). After quality screening, 218 environments and 1,055 tasks are retained, further supporting the validity and reliability of OL-Bench.

Overall, OL-Bench combines real-world service grounding, executable operational workflows, and structured human validation to support its scenario representativeness and environment validity. We have expanded the discussion of real-world grounding in Section 3.1 and provided detailed scenario coverage statistics and representative workflow mappings in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer fSHU (3/3)

Overall Contributions.

We would also like to emphasize that OL-Bench contributes beyond simply expanding scenario coverage. Specifically, we introduce (1) an Entity–Tool–Instruction hierarchical framework for scalable scenario and task synthesis, (2) a large-scale interactive benchmark comprising 218 scenarios and 1,055 tasks, together with 18 fine-grained metrics across seven dimensions for assessing agent behavior throughout multi-turn interactions, and (3) systematic evaluations of seven representative models that reveal capability limitations and common failure patterns. Together, these contributions provide a structured framework for synthesizing complex interactive tasks and understanding agent capabilities in real-life scenarios.


Thank you once again for your valuable feedback and for recognizing our environment and task generation pipeline and interesting analyses of model strengths, weaknesses, and failure modes.

We are also encouraged by the positive assessments from Reviewers 6sfW and GtQn regarding:

  1. Benchmark Coverage: (6sfW: Broad benchmark coverage; GtQn: Broad, structured coverage of interactive agent behavior).
  2. Framework Design: (6sfW: Systematic task generation, dependency-aware task graphs, and L1/L2/L3 distractors).
  3. Evaluation and Analysis: (6sfW: Fine-grained evaluation and useful empirical observations; GtQn: Fine-grained evaluation with an explicit reliability effort).

We hope our responses have satisfactorily addressed your concerns and welcome further discussion. If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.

Reference:

[1] Meituan, Service Retail Business Categories, Official Platform Documentation, 2025. https://rules-center.meituan.com/m/detail/guize/318?activeRule=1

[2] Alibaba Group, Taobao: Business Overview, Official Website. https://home.alibabagroup.com/en-US/about-alibaba-businesses-1744508703945523200

[3] Trip.com Group, Our Brands, Official Website. https://group.trip.com/brands

[4] China Railway, 12306 Official Railway Ticketing Platform, Official Website. https://www.12306.cn/en/

[5] LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services, KDD 2026.

[6] VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications, ICLR 2026.

[7] TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios, ICML 2026.

Reviewer_GtQnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OL-Bench, a benchmark for evaluating language-model agents in multi-turn, tool-using scenarios intended to resemble real-life services. The benchmark construction pipeline uses an Entity-Tool-Instruction hierarchy: entities define executable world states, tools query or modify those states, and system prompts plus dynamically loadable Skills specify operational rules and workflows. It contains 1,055 tasks across 218 scenarios, with four difficulty levels, dependency-aware task graphs, planned valid answers, and three levels of distractors. A scenario-specific harness coordinates the simulated user, tool execution, state transitions, and trajectory recording. Evaluation uses 18 metrics grouped into seven dimensions, including full and partial task completion, constraint adherence, intent understanding, tool use, factuality, behavioral consistency, and efficiency. Seven models are evaluated in a non-thinking setting, with three also evaluated in a thinking setting. GPT-5.6-Sol obtains the highest Full Task Completion Rate at 18.0%. The paper further reports performance by task complexity, a failure taxonomy, human agreement with the LLM judge, and human ratings of the user simulator.

Critical strengths
  1. Broad, structured coverage of interactive agent behavior. OL-Bench goes beyond isolated function calling by combining stateful environments, progressively disclosed requirements, dependency-aware subtasks, dynamic Skill loading, tool-interface perturbations, and controlled distractors. Table 2 indicates nontrivial task structure, including an average of 5.1 required tools and 4.2 required Skills per task. This breadth makes the benchmark potentially useful for identifying failures that final-answer-only evaluations miss.

  2. Fine-grained evaluation with an explicit reliability effort. The 18 metrics distinguish completion from intent understanding, constraint adherence, tool use, factuality, consistency, and efficiency. Section 7 evaluates 300 trajectories with three human annotators and reports 95.3% raw agreement and Cohen's kappa of 0.84 for Judge-based items, along with human ratings of simulator fidelity and persona consistency. This is a stronger validation effort than is typical for fully automated benchmark pipelines, even though additional stratification is needed.

Critical weaknesses
  1. The validity and reproducibility of the synthetic benchmark artifacts are not adequately established. Nearly every component - entity data, tool implementations, Skills, system instructions, task graphs, distractors, checklists, user behavior, and most evaluation signals - is generated or mediated by LLMs. Section 3.2 reports manual inspection of 40 environments and 100 tasks, but does not report the rubric, annotator count, agreement, number or types of errors found, correction rate, or an independent post-correction audit. It is also unclear whether every task has an executable oracle trajectory that completes all checklist items. More broadly, the claim of 218 "real-life" scenarios is supported only by a general statement that designs were informed by platforms and documentation; there is no scenario taxonomy, source-to-scenario mapping, coverage analysis, or human realism comparison against actual user workflows. The example in Appendix A.4 is coherent, but one example cannot validate 1,055 generated tasks. Most importantly for a benchmark paper, the Reproducibility Statement provides no code, data, executable environments, generation prompts, or artifact link. Without the actual benchmark package, the main contribution and its reported results cannot be independently inspected or reproduced.

  2. The evaluation design leaves unresolved judge/simulator bias and statistical uncertainty. Table 4 shows that 16 of the 18 metrics, including FTCR and SCR, rely on a GPT-5.6-Sol Judge, while GPT-5.6-Sol is also the top-ranked evaluated model. The user simulator and all benchmark construction are based on DeepSeek-V4-Flash, which is likewise one of the evaluated models. These overlaps create plausible self-family or style compatibility effects. The aggregate reliability numbers in Section 7 do not resolve this concern because agreement is not broken down by metric, evaluated model, difficulty, positive/negative label, or judge-target identity; pooled agreement can also conceal errors in the all-items-must-pass FTCR calculation. In addition, Section 5.1 runs only one trajectory per model-task pair, reports no confidence intervals or significance tests, and omits essential generation settings such as sampling parameters, exact reasoning controls, turn/tool-call limits, and context-compression settings. Since several FTCR differences are only a few percentage points and interaction trajectories involve a generative user simulator, the stability of the ranking and thinking-versus-non-thinking conclusions is unknown.

Questions for authors
  1. Will the authors release the complete benchmark package, including all 1,055 task specifications, entities, executable tools, Skills, system prompts, user profiles, checklists, generation prompts, harness code, model outputs, and evaluation code? Can every task be passed by a human-authored or programmatically constructed oracle trajectory, and what fraction was independently audited after all corrections?

  2. Can the Judge validation be reported separately by each of the 16 Judge-based metrics, evaluated model, difficulty level, and pass/fail class, including precision, recall, confusion matrices, and agreement specifically on FTCR? Does GPT-5.6-Sol remain the top model when evaluated by human labels or by an independent judge model? Similarly, does changing the DeepSeek-V4-Flash user simulator affect DeepSeek-family versus non-DeepSeek model rankings?

  3. Can the authors provide uncertainty estimates and repeat a representative subset with multiple trajectories or simulator seeds? Please also specify endpoint versions, temperatures/sampling settings, maximum output and reasoning budgets, maximum interaction turns and tool calls, context-compression policy, and how the "thinking" and "non-thinking" settings are made comparable.

7 comments
AuthorsOpenReview ↗

Response to Reviewer GtQn (1/4)

Dear reviewer, thank you very much for your time and effort in reviewing our paper. We try to address your concerns as follows and hope we can ease your concerns.


Q1: How are the validity and solvability of LLM-synthesized environments and tasks ensured?

A1: OL-Bench ensures the reliability of synthesized environments and tasks through deterministic execution checks, answer-grounded task construction, and iterative human validation, rather than relying solely on LLM judgments.

(1) Environment Validity. Each synthesized environment undergoes deterministic execution tests to verify tool functionality, entity consistency, and environment-state updates. Environments failing these checks are rejected.

(2) Task Solvability. We first define the answer requirements for each subtask and then generate entities satisfying these requirements, ensuring the existence of valid answers by construction. Automated checks further verify data consistency, entity relationships, and tool support, rejecting tasks that fail validation.

(3) Iterative Human Quality Validation. We adopt a multi-round generation → inspection → error analysis → framework refinement → regeneration → revalidation process. In each round, identified issues are addressed at the synthesis-framework level, followed by random resampling from regenerated environments and tasks, rather than merely rechecking previously identified errors. Across this process, five annotators inspect 40 environments and 100 tasks, with each sample independently reviewed by three annotators, achieving 91.2% agreement (Fleiss' κ = 0.82). We identify and correct 23 errors, incorporating the improvements into subsequent environment and task generation.

(4) Full-Scale Human Audit. We conduct a comprehensive human audit of the generated candidate environments and tasks, ultimately retaining 218 environments and 1,055 tasks after quality screening. Six annotators are divided into two independent groups of three, with each group reviewing all candidates without intergroup communication. The audit covers entity and data consistency, tool correctness, task executability, and Checklist validity. The two groups achieve 97.8% agreement (Cohen's κ = 0.94). For samples with conflicting assessments, all six annotators jointly review the cases, identify potential issues, and correct or discard invalid environments and tasks.

Together, these mechanisms provide complementary validation of environment executability, task feasibility, and synthesized artifact quality.

We have added the detailed validation procedures and human audit results to Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GtQn (2/4)

Q2: How does OL-Bench establish real-world representativeness and validate the realism of its synthesized environments?

A2: (1) Real-World Scenario Coverage. OL-Bench is grounded in real-world service platforms, public documentation, and operational experience from a deployed industrial agent for local-life services. Statically, real user requests inform task categories, difficulty levels, and query styles. Dynamically, industrial agent practices guide the design of Skills, Harness, and tool execution mechanisms. Based on these references, OL-Bench covers 12 service categories, 218 environments, and 1,055 tasks, spanning representative services from Meituan, Dianping, Taobao, JD.com, Ctrip, and 12306 [1–4], as detailed below.

CategoryRepresentative PlatformsEnvsTasks
TransportationCtrip, 1230630134
HealthcareMeituan, JD.com20132
AccommodationCtrip, Meituan1689
Leisure & EntertainmentMeituan, Dianping2188
RetailTaobao, JD.com2082
Daily-Life ServicesMeituan1571
Beauty & WellnessMeituan, Dianping1566
Pet ServicesMeituan1457
Food & DeliveryMeituan1154
FitnessMeituan629
Education & FamilyMeituan429
Composite ScenariosCross-Domain Services46224
Total12 Categories2181,055

Compared with LocalSearchBench, VitaBench, and TRIP-Bench [5-7], OL-Bench provides complementary coverage of pet services, beauty and wellness, fitness, education, and cross-domain interactions, including 46 composite environments and 224 tasks.

(2) Environment Realism. The complexity of real-world service systems primarily involves four aspects: (i) Task Dependencies, where operations follow interdependent workflows; (ii) Entity Complexity, requiring multi-constraint filtering among interconnected candidates; (iii) Business Rules, governing operational requirements and prerequisites; and (iv) State Transitions, where actions modify environment states and affect subsequent operations.

As described in Section 3 (Benchmark Construction) of our paper, OL-Bench models these complexities through the Entity–Tool–Instruction hierarchy, which defines interconnected entities, executable tools, and operational rules. Information Graphs capture task dependencies, L1/L2/L3 distractors introduce constraint-based selection challenges, and the Agent Harness manages tool execution and state updates.

(3) Human Validation. As detailed in Q1, we conduct iterative human validation on 40 environments and 100 tasks, achieving 91.2% agreement (Fleiss' κ = 0.82). We further perform a full-scale human audit using two independent groups of three annotators, achieving 97.8% agreement (Cohen's κ = 0.94). After quality screening, 218 environments and 1,055 tasks are retained, further supporting the validity and reliability of OL-Bench.

Overall, OL-Bench combines real-world service grounding, executable operational workflows, and structured human validation to support its scenario representativeness and environment validity. We have expanded the discussion of real-world grounding in Section 3.1 and provided detailed scenario coverage statistics and representative workflow mappings in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer GtQn (3/4)

Q3: Will the complete OL-Bench implementation and evaluation artifacts be publicly available to ensure reproducibility?

A3: We have provided the complete OL-Bench implementation in the supplementary materials, including: (1) 218 environments and 1,055 tasks with their configurations; (2) the Entity–Tool–Instruction-based environment and task synthesis pipeline; (3) environment state management, tool definitions, and execution logic; (4) the unified Agent Harness for Agent–Environment interactions, tool execution, state updates, and trajectory recording; and (5) the evaluation code, including Checklists, metric computation, and LLM Judge prompts and configurations.


Q4: Do overlapping roles of evaluated models, Judges, and user simulators introduce evaluation bias?

A4: To investigate potential evaluation bias, we conduct cross-Judge evaluation and user-simulator replacement experiments.

(1) Cross-Judge Evaluation. We re-evaluate 1,055 tasks × 7 non-thinking models using GPT-5.6-Sol, DeepSeek-V4-Flash, and GLM-5.2 as Judges, under identical Checklists and protocols.

Evaluation MetricResult
Checklist Exact Agreement81.1%
Ranking Consistency (Kendall's W)0.862
GPT-5.6-Sol Top-3 Probability (GPT / Flash / GLM Judge)100% / 100% / 96.8%

The results show substantial agreement across Judges, with GPT-5.6-Sol maintaining strong performance under independent evaluation rather than only self-evaluation.

(2) User-Simulator Replacement. We replace DeepSeek-V4-Flash with GLM-5.1 and evaluate 100 tasks × 7 models, keeping environments, tools, Agent Harness, user profiles, and Judge unchanged.

Evaluation MetricResult
Ranking Consistency (Spearman)0.893
Ranking Consistency (Kendall's W)0.946
Mean ΔFTCR: DeepSeek Models−0.5 pp
Mean ΔFTCR: Non-DeepSeek Models−1.8 pp
Between-Group Difference (Non-DeepSeek − DeepSeek)−1.3 pp

Here, ΔFTCR denotes the change after simulator replacement. The 95% confidence interval for the between-group difference includes zero, indicating no statistically conclusive evidence of a systematic DeepSeek-family advantage.

Overall, these experiments show substantial cross-Judge agreement and broadly consistent model rankings under simulator replacement, providing additional evidence against systematic biases arising from overlapping model roles.

We have added the above analyses and results to Appendices B and C of the revised version.

Reviewer_6sfWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces OL-Bench, a benchmark for evaluating AI agents across 218 real-life-inspired scenarios and 1,055 interactive tasks. It proposes an Entity–Tool–Instruction hierarchy for synthesizing executable environments, dependency-aware tasks with controllable difficulty, and 18 evaluation metrics across seven capability dimensions. Experiments across seven models show relatively low end-to-end task completion (best FTCR of 18%), with retrieval and constraint filtering accounting for most reported failures.

The paper presents a potentially useful benchmark construction framework with broad synthetic scenario coverage and fine-grained evaluation. However, several important aspects remain insufficiently validated, particularly the real-world representativeness of the synthesized environments, the agent runtime used for evaluation, and the extent to which the benchmark measures capabilities beyond retrieval and constraint satisfaction.

The heavy dependence on LLM-based judging, absence of critical ablations, and limited comparative validation further weaken the empirical claims. While the synthesis methodology is promising, the paper does not convincingly establish that OL-Bench provides a more realistic, comprehensive, or reproducible evaluation of agent capabilities than existing benchmarks.

Critical strengths
  • Broad benchmark coverage: The benchmark spans diverse scenarios, tools, workflows, and task complexities, providing a potentially useful framework for evaluating interactive agents.

  • Systematic task generation: The structured synthetic-world generation pipeline, dependency-aware task graphs, and L1/L2/L3 distractors offer a scalable approach to constructing evaluation environments.

  • Fine-grained evaluation: The 18 metrics go beyond final task completion and attempt to characterize agent capabilities and failure modes across multiple dimensions.

  • Useful empirical observations: The analysis highlights the gap between subtask and end-to-end completion, as well as the degradation in performance with increasing task complexity.

Critical weaknesses
  • Real-world coverage and validity: While the paper claims comprehensive coverage of real-life scenarios, the environments and tasks are synthetically generated. It is unclear whether the resulting environments reproduce the operational complexity, state dependencies, and constraints of real business systems. Diversity across synthetic scenarios does not necessarily establish broad real-world coverage.

  • Unclear agent runtime and experimental setup: The paper introduces a scenario-specific harness but does not adequately describe the agent framework or runtime used to execute the evaluated models. Specifically, it is unclear who implements the reasoning–action–observation loop, how tool execution is orchestrated, and how retries, tool errors, context management, and stopping conditions are handled. Without these details, it is difficult to attribute the observed failures to the underlying models versus the surrounding agent system.

  • Limited validation of task difficulty: Difficulty is controlled through entity scale, tool requirements, and dependency-chain complexity. However, it is unclear whether increasing difficulty measures deeper reasoning and planning capabilities or primarily increases the number of opportunities for errors, thereby reducing end-to-end task completion.

  • Missing ablations: The L1/L2/L3 distractor mechanism is a useful design for testing constraint satisfaction, but the contributions of individual distractor categories are not evaluated. Similarly, the impact of tool perturbations, Skills, and dependency-aware task construction is not independently established.

  • Heavy reliance on LLM-based evaluation: Of the 18 metrics, only interaction turns and token consumption are explicitly deterministic. The remaining 16 rely on an LLM judge, including checklist-based task completion. Furthermore, GPT-5.6-Sol serves as both the judge and the best-performing evaluated model, raising concerns about potential evaluation bias. The use of an LLM judge for outcomes that may be verifiable through deterministic environment-state checks also deserves justification.

  • Limited failure analysis: Approximately 95.5% of reported failures involve retrieval/query strategies and insufficient filtering/verification. This raises questions about whether the benchmark primarily evaluates search and constraint satisfaction rather than the broader spectrum of real-world agent capabilities. The paper also does not sufficiently explain how failure categories are annotated, how overlapping failures are handled, or whether the observed distribution is influenced by the benchmark's construction choices. No causal analysis establishes whether addressing these failures improves end-to-end success.

  • Benchmark comparison and reproducibility: Table 1 presents OL-Bench as substantially more comprehensive than existing benchmarks, but some comparison criteria appear tailored to the proposed framework. For example, established benchmarks such as τ-Bench and τ²-Bench have executable evaluation infrastructure yet are marked as lacking harness support. While OL-Bench offers a potentially valuable synthesis methodology, the paper does not demonstrate comparable reproducibility, independent adoption, or empirical advantages over established benchmarks. The availability of the complete implementation and evaluation artifacts is also unclear.

Questions for authors
  • Real-world representativeness: Does the paper establish broad real-world coverage, or primarily demonstrate diversity within synthetically generated tasks? How have the authors validated that these environments reproduce the complexity of real business systems?

  • Agent runtime: What agent framework or runtime is used to execute the evaluated models? Who implements the reasoning–action–observation loop? Is this a standardized ReAct implementation or a custom agent controller? How are retries, tool errors, context management, and stopping conditions implemented and controlled across models?

  • Failure attribution: Given that approximately 95.5% of failures involve retrieval and filtering, to what extent does OL-Bench evaluate broader agent capabilities such as long-horizon planning, state management, error recovery, and execution reliability? Could this failure distribution be driven by the benchmark's construction methodology?

  • Difficulty calibration: How do the authors distinguish increased reasoning and planning difficulty from the compounding probability of failure as the number of required tools, subtasks, and constraints increases?

  • Ablation studies: Can the authors provide ablations isolating the effects of L1/L2/L3 distractors, tool perturbations, Skills, and dependency-graph complexity on benchmark performance?

  • Evaluation reliability: Why are checklist-based completion and environment-state correctness not evaluated deterministically wherever feasible? Have the authors assessed judge bias, particularly given that GPT-5.6-Sol is both an evaluated model and the judge?

  • Failure taxonomy: How are failure categories assigned? Are multiple failure modes allowed per trajectory? What is the inter-annotator agreement, and has the taxonomy been validated against independent human assessment?

  • Comparison with existing benchmarks: How are the criteria in Table 1 defined, particularly harness support? Can the authors demonstrate that OL-Bench captures additional capabilities or provides diagnostic value beyond τ-Bench, τ²-Bench, and VitaBench through direct empirical comparisons?

  • Reproducibility: Will the complete benchmark, environment and task synthesis pipeline, agent runtime, tool implementations, and evaluation code be made publicly available?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 6sfW (1/5)

Dear reviewer, thank you very much for your time and effort in reviewing our paper. We try to address your concerns as follows and hope we can ease your concerns.


Q1: How does OL-Bench establish real-world representativeness and validate the realism of its synthesized environments?

A1: (1) Real-World Scenario Coverage. OL-Bench is grounded in real-world service platforms, public documentation, and operational experience from a deployed industrial agent for local-life services. Statically, real user requests inform task categories, difficulty levels, and query styles. Dynamically, industrial agent practices guide the design of Skills, Harness, and tool execution mechanisms. Based on these references, OL-Bench covers 12 service categories, 218 environments, and 1,055 tasks, spanning representative services from Meituan, Dianping, Taobao, JD.com, Ctrip, and 12306 [7–10], as detailed below.

CategoryRepresentative PlatformsEnvsTasks
TransportationCtrip, 1230630134
HealthcareMeituan, JD.com20132
AccommodationCtrip, Meituan1689
Leisure & EntertainmentMeituan, Dianping2188
RetailTaobao, JD.com2082
Daily-Life ServicesMeituan1571
Beauty & WellnessMeituan, Dianping1566
Pet ServicesMeituan1457
Food & DeliveryMeituan1154
FitnessMeituan629
Education & FamilyMeituan429
Composite ScenariosCross-Domain Services46224
Total12 Categories2181,055

Compared with LocalSearchBench, VitaBench, and TRIP-Bench [1, 5, 6], OL-Bench provides complementary coverage of pet services, beauty and wellness, fitness, education, and cross-domain interactions, including 46 composite environments and 224 tasks.

(2) Environment Realism. The complexity of real-world service systems primarily involves four aspects: (i) Task Dependencies, where operations follow interdependent workflows; (ii) Entity Complexity, requiring multi-constraint filtering among interconnected candidates; (iii) Business Rules, governing operational requirements and prerequisites; and (iv) State Transitions, where actions modify environment states and affect subsequent operations.

As described in Section 3 (Benchmark Construction) of our paper, OL-Bench models these complexities through the Entity–Tool–Instruction hierarchy, which defines interconnected entities, executable tools, and operational rules. Information Graphs capture task dependencies, L1/L2/L3 distractors introduce constraint-based selection challenges, and the Agent Harness manages tool execution and state updates.

(3) Full-Scale Human Audit. We conduct a comprehensive human audit of the generated candidate environments and tasks, ultimately retaining 218 environments and 1,055 tasks after quality screening. Six annotators are divided into two independent groups of three, with each group reviewing all candidates without intergroup communication. The audit covers entity and data consistency, tool correctness, task executability, and Checklist validity. The two groups achieve 97.8% agreement (Cohen's κ = 0.94). For samples with conflicting assessments, all six annotators jointly review the cases, identify potential issues, and correct or discard invalid environments and tasks.

Overall, OL-Bench combines real-world service references, executable operational workflows, and structured human validation to support the grounding and validity of its synthesized environments. We have expanded the discussion of real-world grounding in Section 3.1 and provided detailed scenario coverage statistics and representative workflow mappings in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer 6sfW (2/5)

Q2: How does OL-Bench distinguish reasoning difficulty from error accumulation caused by additional subtasks?

A2: We agree that lower task completion rates may result from both error accumulation and increased dependency complexity. To distinguish these factors, we conduct a two-way stratified analysis using the average task completion rate across seven evaluated models under the non-thinking setting. Specifically, we examine (1) the effect of subtask count at fixed dependency depth and (2) the effect of dependency depth at fixed subtask count, using linear regression within each group.

Controlled VariableFixed ValueFTCR Change per Unit Increase
Dependency Depth2 / 3 / 4 / 5−3.46 / −4.24 / −2.69 / −1.54 pp
Subtask Count4 / 5 / 6−2.10 / −5.03 / −1.88 pp

The results show that each additional subtask is associated with a 1.54–4.24 pp decrease in FTCR at fixed dependency depth. More importantly, each additional dependency level is associated with a 1.88–5.03 pp decrease even at fixed subtask count. These findings suggest that OL-Bench difficulty reflects not only the number of subtasks but also the complexity of task dependencies.


Q3: What Agent runtime is used in OL-Bench, and how are execution mechanisms controlled across models?

A3: OL-Bench employs a unified custom Agent runtime with a ReAct-style controller. The runtime manages the reasoning–action–observation loop, tool execution, context management, and termination, while each environment configures its own tools, states, Skills, and execution parameters. We clarify the implementation details below.

(1) Agent Runtime. The reasoning–action–observation loop is implemented by ReactLoop.run(). At each step, the model receives the conversation history and available tool schemas. Tool calls are handled by ToolExecutor, with results appended as observations. The loop continues until the model produces a final answer or a termination condition is triggered.

(2) Tool Execution and Error Handling. ToolExecutor executes tools in-process, supporting both environment-state queries and modifications. Tool execution errors are returned as observations, allowing the Agent to determine subsequent actions. The runtime enforces configurable tool-call and output-length limits and supports dynamic Skill loading through load_skills.

(3) Context Management and Termination. All models share the same context-management module, which applies LLM-based summarization or rule-based truncation at predefined thresholds while retaining recent interactions and tool calls.

Termination conditions include final answers, max_rounds/max_traces limits, API retry exhaustion, empty-response retry exhaustion, user-simulator stop signals, and max_turns. Upon reaching an execution limit, the controller explicitly requests a final report and records forced_answer.

(4) Consistency Across Models. For fair comparison, all evaluated models use identical runtime code, tool execution mechanisms, and execution configurations within each environment, with only the evaluated model replaced. The user simulator is consistently implemented using DeepSeek-V4-Flash. Configurations may vary across environments but remain identical across models evaluated in the same environment, minimizing runtime-related confounding factors.

We have added these clarifications in Section 3.3 of the revised version.


Q4: Why does OL-Bench rely on LLM-based evaluation instead of deterministic checks?

A4: We adopt LLM-based evaluation because many task requirements cannot be verified solely through environment states. Following VitaBench[1] and OmniaBench[2], our Judge evaluates Checklist items using execution trajectories, tool outputs, and environment states.

(1) Evaluation Design. Among 4,960 Checklist items, 33.5% involve state-changing operations that are potentially verifiable through deterministic state checks, while the remaining 66.5% primarily involve information retrieval, constraint filtering, and user interactions requiring trajectory-level or semantic evaluation. Moreover, the other 14 LLM-judged metrics involve semantic judgments. Even for state-verifiable items, deterministic checks require task-specific rules (e.g., booking details versus coupon status), making a unified implementation across 218 environments challenging.

(2) Deterministic Validation. We agree that deterministic checks are preferable when applicable. We therefore compare LLM judgments with programmatic state checks on 50 verifiable requirements from a representative environment, achieving 94.0% agreement and Cohen's κ = 0.876. These results support the reliability of LLM-based evaluation for state-verifiable requirements.

AuthorsOpenReview ↗

Response to Reviewer 6sfW (3/5)

Q5: How does OL-Bench assess potential Judge bias, particularly when GPT-5.6-Sol serves as both an evaluated model and the Judge?

A5: To assess Judge reliability and potential self-evaluation bias, we re-evaluated the trajectories of 1,055 tasks across seven non-thinking models using three different Judges: GPT-5.6-Sol, DeepSeek-V4-Flash, and GLM-5.2, under identical Checklists and evaluation protocols. We examine Checklist-level agreement, model score and ranking consistency, and ranking stability.

Evaluation AspectMetricResult
Checklist AgreementThree-Judge Exact Agreement81.1%
Ranking ConsistencyKendall's W0.862
Score ConsistencyICC(2,k) / ICC(3,k)0.868 / 0.872
Ranking CorrelationMean Pairwise Spearman0.793
GPT-5.6-Sol Top-3 StabilityBootstrap (GPT / Flash / GLM Judge)100% / 100% / 96.8%

(1) Checklist Agreement. The three Judges achieve 81.1% exact agreement on individual Checklist items, indicating that most judgments remain consistent across different evaluation models.

(2) Score and Ranking Consistency. The high Kendall's W (0.862), mean pairwise Spearman correlation (0.793), and ICC scores (0.868/0.872) indicate substantial agreement in model rankings and scores across Judges.

(3) Potential Self-Evaluation Bias. Bootstrap analysis shows that GPT-5.6-Sol ranks among the top three models with probabilities of 100%, 100%, and 96.8% under GPT-5.6-Sol, DeepSeek-V4-Flash, and GLM-5.2 Judges, respectively.

Overall, these results indicate consistent evaluation outcomes across different Judges, with GPT-5.6-Sol maintaining competitive performance when evaluated by other models.

We have added these analyses and results in Appendix B of the revised version.


Q6: What are the individual effects of L1/L2/L3 distractors, tool perturbations, Skills, and dependency-graph complexity on benchmark performance?

A6: To investigate the contributions of different benchmark mechanisms, we conduct additional ablation experiments and dependency-depth analysis.

(1) Experimental Setup. We select 50 tasks and seven non-thinking models and construct environment variants by progressively removing L1/L2/L3 distractors, disabling tool perturbations, or disabling Skill Hiding. All experiments use the same Agent runtime, with DeepSeek-V4-Flash as the user simulator and GPT-5.6-Sol as the Judge, enabling paired comparisons across the same tasks and models.

(2) Ablation Results.

SettingFTCR (%)SCR (%)ΔFTCR (pp)ΔSCR (pp)
Original (C0)10.648.7——
Remove L112.454.5+1.8+5.8
Remove L1+L214.655.9+4.0+7.2
Remove L1+L2+L315.456.1+4.8+7.4
Disable Tool Perturbation12.050.7+1.4+2.0
Disable Skill Hiding17.755.8+7.1+7.1

Removing L1 distractors improves FTCR by 1.8 pp, while further removing L2 and L3 increases the cumulative improvement to 4.0 pp and 4.8 pp, respectively, relative to C0. Disabling tool perturbations improves FTCR by 1.4 pp, while disabling Skill Hiding yields the largest FTCR improvement (+7.1 pp), highlighting the challenge of dynamic Skill discovery and loading.

(3) Dependency-Graph Complexity. As demonstrated in our response to Q2, increasing dependency depth is associated with a 1.88–5.03 pp decrease in FTCR per additional level, even when the number of subtasks is held fixed. This indicates that task dependency structure contributes to difficulty beyond subtask count alone.

Overall, these results show that distractors, tool perturbations, and Skill Hiding affect Agent performance, while dependency complexity is also associated with lower task completion rates.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank you for your constructive feedback and for recognizing the strengths of OL-Bench, including its broad and structured benchmark coverage, systematic environment and task generation, fine-grained evaluation framework, and useful empirical analyses. Your valuable suggestions have helped us further strengthen the experimental validation and methodological clarity of our work.

Below, we summarize the additional experiments, clarifications, and manuscript revisions addressing the concerns raised by Reviewers 6sfW, GtQn, and fSHU.

Additional Experiments.

  1. Task Difficulty Analysis. Separately analyzed the effects of subtask count and dependency depth through controlled stratified analyses. (Reviewer 6sfW)
  2. Ablation Studies. Evaluated the effects of L1/L2/L3 distractors, tool perturbations, and Skill Hiding. (Reviewer 6sfW)
  3. Deterministic Evaluation Validation. Compared LLM judgments with programmatic checks on 50 verifiable requirements, achieving 94.0% agreement (κ = 0.876). (Reviewer 6sfW)
  4. Cross-Judge Reliability. Re-evaluated 1,055 tasks × 7 models using three Judges, obtaining 81.1% Checklist agreement, Kendall's W = 0.862, and ICC = 0.868/0.872. Added to Appendix B. (Reviewers 6sfW, GtQn)
  5. User-Simulator Sensitivity. Replaced DeepSeek-V4-Flash with GLM-5.1 on 100 tasks × 7 models, obtaining Spearman = 0.893 and Kendall's W = 0.946. Added to Appendix C. (Reviewer GtQn)
  6. Human–Judge and FTCR Validation. Analyzed 300 human-annotated trajectories, obtaining 95.3% Checklist-level agreement, 97.0% task-level agreement, and a 1.0 pp FTCR difference. Added to Appendix D. (Reviewer GtQn)
  7. Repeated-Run Stability. Conducted three runs on 500 tasks × 7 models (10,500 trajectories), yielding identical model rankings (Spearman = 1.000) and approximately 0.97 pp run-to-run FTCR standard deviation. (Reviewer GtQn)
  8. Broader Capability and Failure Analysis. Examined multi-step planning, state-changing operations, and invalid-path handling, with 94.6% agreement between human annotations and failure classifications. (Reviewer 6sfW)

Additional Clarifications.

  1. Real-World Grounding. Clarified service-platform references, industrial Agent practices, and coverage of 12 categories, 218 environments, and 1,055 tasks. Revised Section 3.1 and Appendix A. (Reviewers 6sfW, GtQn, fSHU)
  2. Environment and Task Validity. Clarified deterministic execution checks, answer-grounded task construction, and iterative human validation (40 environments and 100 tasks; 91.2% agreement, κ = 0.82). We further described the full-scale human audit, involving two independent groups of three annotators and quality screening of candidate environments and tasks. Detailed validation procedures and audit results were added to Appendix F. (Reviewers GtQn, fSHU)
  3. Agent Runtime and Harness. Clarified the ReAct-style runtime, tool execution, context management, termination conditions, and cross-model consistency in Section 3.3. (Reviewer 6sfW)
  4. Evaluation Design. Explained the use of LLM Judges and the limitations of deterministic state verification. (Reviewer 6sfW)
  5. Experimental Reproducibility. Detailed model settings, Harness configurations, context compression, and Thinking/Non-thinking settings in Appendix E. (Reviewer GtQn)
  6. Code and Data Availability. Provided the OL-Bench implementation and dataset in the supplementary materials, including 218 environments, 1,055 tasks, the synthesis pipeline, Agent Harness, and evaluation code. (Reviewers 6sfW, GtQn)

Manuscript Revisions.

  1. Expanded Section 3.1 and Appendix A on real-world grounding and scenario coverage.
  2. Clarified the unified Agent runtime in Section 3.3.
  3. Added cross-Judge, user-simulator, and human–Judge reliability analyses in Appendices B–D.
  4. Added model and Harness configurations in Appendix E.
  5. Added detailed environment and task validation procedures, iterative human quality control, and full-scale human auditing in Appendix F.
  6. Corrected the Harness Support entries in Table 1 and clarified the comparison criteria for existing benchmarks.

Outcome & Score.

Before Nov 18, Reviewers 6sfW and GtQn acknowledged our responses and both raised their scores to 3. Reviewer fSHU has not engaged in the discussion.

Overall, we believe these additional experiments, methodological clarifications, and manuscript revisions strengthen the validity, reliability, and reproducibility of OL-Bench. We hope this summary facilitates the assessment of our rebuttal and revised manuscript.

Thank you once again for your dedication to the community.

Best regards,

The Authors

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.