- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents OL-Bench, an interactive benchmark for agents in diverse, real-life scenarios. This benchmark synthesizes environments from an Entity-Tool-Instruction hierarchy. Environments have scenario-specific skills, tool perturbations introduce variations in tools, and dependency-aware multi-turn tasks across different difficulty levels. Additionally, they introduce an interaction evaluation framework with 18 metrics across 7 dimensions. The authors evaluate several models and show that their benchmark presents a challenge for current agents.
Critical strengths
The authors present a pipeline for generating realistic environments and tasks to evaluate agents in diverse and realistic scenarios. When evaluating they present interesting analyses, about how different models have different strengths/weaknesses and why models fail.
Critical weaknesses
This paper presents a pipeline for environment and task synthesis, and demonstrates that models achieve an 18% task completion rate. The authors randomly sample 40 environments and 100 tasks for manual inspection, and correct any issues they identify. Beyond that, there doesn’t seem to be much analysis of whether the tasks and environments they synthesize are valid or truly solvable.
One claim of the paper is that the benchmark presented has diverse and realistic scenarios. However, there is no analysis that proves this claim. The authors can consider presenting metrics/analyses that compare the diversity and realism of their benchmark with existing benchmarks.
Questions for authors
n/a
5 comments
Response to Reviewer fSHU (2/3)
Q2: How does OL-Bench establish real-world representativeness and validate the realism of its synthesized environments?
A2: (1) Real-World Scenario Coverage. OL-Bench is grounded in real-world service platforms, public documentation, and operational experience from a deployed industrial agent for local-life services. Statically, real user requests inform task categories, difficulty levels, and query styles. Dynamically, industrial agent practices guide the design of Skills, Harness, and tool execution mechanisms. Based on these references, OL-Bench covers 12 service categories, 218 environments, and 1,055 tasks, spanning representative services from Meituan, Dianping, Taobao, JD.com, Ctrip, and 12306 [1–4], as detailed below.
| Category | Representative Platforms | Envs | Tasks |
|---|---|---|---|
| Transportation | Ctrip, 12306 | 30 | 134 |
| Healthcare | Meituan, JD.com | 20 | 132 |
| Accommodation | Ctrip, Meituan | 16 | 89 |
| Leisure & Entertainment | Meituan, Dianping | 21 | 88 |
| Retail | Taobao, JD.com | 20 | 82 |
| Daily-Life Services | Meituan | 15 | 71 |
| Beauty & Wellness | Meituan, Dianping | 15 | 66 |
| Pet Services | Meituan | 14 | 57 |
| Food & Delivery | Meituan | 11 | 54 |
| Fitness | Meituan | 6 | 29 |
| Education & Family | Meituan | 4 | 29 |
| Composite Scenarios | Cross-Domain Services | 46 | 224 |
| Total | 12 Categories | 218 | 1,055 |
Compared with LocalSearchBench, VitaBench, and TRIP-Bench [5-7], OL-Bench provides complementary coverage of pet services, beauty and wellness, fitness, education, and cross-domain interactions, including 46 composite environments and 224 tasks.
(2) Environment Realism. The complexity of real-world service systems primarily involves four aspects: (i) Task Dependencies, where operations follow interdependent workflows; (ii) Entity Complexity, requiring multi-constraint filtering among interconnected candidates; (iii) Business Rules, governing operational requirements and prerequisites; and (iv) State Transitions, where actions modify environment states and affect subsequent operations.
As described in Section 3 (Benchmark Construction) of our paper, OL-Bench models these complexities through the Entity–Tool–Instruction hierarchy, which defines interconnected entities, executable tools, and operational rules. Information Graphs capture task dependencies, L1/L2/L3 distractors introduce constraint-based selection challenges, and the Agent Harness manages tool execution and state updates.
(3) Human Validation. As detailed in Q1, we conduct iterative human validation on 40 environments and 100 tasks, achieving 91.2% agreement (Fleiss' κ = 0.82). We further perform a full-scale human audit using two independent groups of three annotators, achieving 97.8% agreement (Cohen's κ = 0.94). After quality screening, 218 environments and 1,055 tasks are retained, further supporting the validity and reliability of OL-Bench.
Overall, OL-Bench combines real-world service grounding, executable operational workflows, and structured human validation to support its scenario representativeness and environment validity. We have expanded the discussion of real-world grounding in Section 3.1 and provided detailed scenario coverage statistics and representative workflow mappings in Appendix A.
Response to Reviewer fSHU (3/3)
Overall Contributions.
We would also like to emphasize that OL-Bench contributes beyond simply expanding scenario coverage. Specifically, we introduce (1) an Entity–Tool–Instruction hierarchical framework for scalable scenario and task synthesis, (2) a large-scale interactive benchmark comprising 218 scenarios and 1,055 tasks, together with 18 fine-grained metrics across seven dimensions for assessing agent behavior throughout multi-turn interactions, and (3) systematic evaluations of seven representative models that reveal capability limitations and common failure patterns. Together, these contributions provide a structured framework for synthesizing complex interactive tasks and understanding agent capabilities in real-life scenarios.
Thank you once again for your valuable feedback and for recognizing our environment and task generation pipeline and interesting analyses of model strengths, weaknesses, and failure modes.
We are also encouraged by the positive assessments from Reviewers 6sfW and GtQn regarding:
- Benchmark Coverage: (6sfW: Broad benchmark coverage; GtQn: Broad, structured coverage of interactive agent behavior).
- Framework Design: (6sfW: Systematic task generation, dependency-aware task graphs, and L1/L2/L3 distractors).
- Evaluation and Analysis: (6sfW: Fine-grained evaluation and useful empirical observations; GtQn: Fine-grained evaluation with an explicit reliability effort).
We hope our responses have satisfactorily addressed your concerns and welcome further discussion. If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.
Reference:
[1] Meituan, Service Retail Business Categories, Official Platform Documentation, 2025. https://rules-center.meituan.com/m/detail/guize/318?activeRule=1
[2] Alibaba Group, Taobao: Business Overview, Official Website. https://home.alibabagroup.com/en-US/about-alibaba-businesses-1744508703945523200
[3] Trip.com Group, Our Brands, Official Website. https://group.trip.com/brands
[4] China Railway, 12306 Official Railway Ticketing Platform, Official Website. https://www.12306.cn/en/
[5] LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services, KDD 2026.
[6] VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications, ICLR 2026.
[7] TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios, ICML 2026.
Response to Reviewer fSHU (1/3)
Dear reviewer, thank you very much for your time and effort in reviewing our paper. We try to address your concerns as follows and hope we can ease your concerns.
Q1: How are the validity and solvability of LLM-synthesized environments and tasks ensured?
A1: OL-Bench ensures the reliability of synthesized environments and tasks through deterministic execution checks, answer-grounded task construction, and iterative human validation, rather than relying solely on LLM judgments.
(1) Environment Validity. Each synthesized environment undergoes deterministic execution tests to verify tool functionality, entity consistency, and environment-state updates. Environments failing these checks are rejected.
(2) Task Solvability. We first define the answer requirements for each subtask and then generate entities satisfying these requirements, ensuring the existence of valid answers by construction. Automated checks further verify data consistency, entity relationships, and tool support, rejecting tasks that fail validation.
(3) Iterative Human Quality Validation. We adopt a multi-round generation → inspection → error analysis → framework refinement → regeneration → revalidation process. In each round, identified issues are addressed at the synthesis-framework level, followed by random resampling from regenerated environments and tasks, rather than merely rechecking previously identified errors. Across this process, five annotators inspect 40 environments and 100 tasks, with each sample independently reviewed by three annotators, achieving 91.2% agreement (Fleiss' κ = 0.82). We identify and correct 23 errors, incorporating the improvements into subsequent environment and task generation.
(4) Full-Scale Human Audit. We conduct a comprehensive human audit of the generated candidate environments and tasks, ultimately retaining 218 environments and 1,055 tasks after quality screening. Six annotators are divided into two independent groups of three, with each group reviewing all candidates without intergroup communication. The audit covers entity and data consistency, tool correctness, task executability, and Checklist validity. The two groups achieve 97.8% agreement (Cohen's κ = 0.94). For samples with conflicting assessments, all six annotators jointly review the cases, identify potential issues, and correct or discard invalid environments and tasks.
Together, these mechanisms provide complementary validation of environment executability, task feasibility, and synthesized artifact quality.
We have added the detailed validation procedures and human audit results to Appendix F of the revised manuscript.