acceptodds
Desk rejected at ICLR 2027

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However, most agent benchmarks still rely on synthetic sandboxes, short-horizon tasks, mock-service APIs, and final-answer checks, leaving open whether agents can complete realistic long-horizon work in the runtimes where they are deployed. This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories. Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls, and runs inside a reproducible Docker container hosting an actual CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to real tools rather than mock services. Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification. Across 36 frontier models, the best, GPT 6 Astra, reaches only 70.1% overall under OpenClaw, only five others exceed 60%, and switching harness alone shifts a single model by up to 18 points. These results show that long-horizon, native-runtime agent evaluation remains a far-from-resolved task for current frontier models. We release the tasks and code to support reproducible evaluation.

Primary area:
datasets and benchmarks
Keywords:
Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N4jXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WildClawBench: 60 human-written, long-horizon agent tasks across six categories. The tasks are bilingual and partly multimodal. Each one runs in Docker under a real CLI harness (e.g., OpenClaw, Claude Code) with real tools, and is graded with rule checks, side-effect audits and an LLM judge. Across 36 models, the best scores 70.1%. Further studies show that harness choice (up to 18 points), time budget and added skills all shift performance substantially. Validation includes three runs per model, a judge-swap study and a small human-agreement check.

Critical strengths
  1. It evaluates the deployed system, widely. Tasks run the real CLI harness in Docker with real tools, and grading assets are mounted only after the agent exits. This is what makes the harness study (Tab. 3) possible: the same model shifts 10–18 points across scaffolds. The time- and thinking-budget results add to this. It's a useful message for the community and my main reason for supporting acceptance.
Critical weaknesses
  1. The key differentiating factor of the dataset is not clear, despite using differentiating label like "native runtime" and "bilingual".

The paper calls OSWorld a synthetic sandbox, but OSWorld runs real apps in real OS VMs. In Tab. 1, Claw-Eval differs from WildClawBench only on "native runtime" (⋄ vs. ✓). WildClawBench's Social tasks also use mocked participants and mock Slack/email tools. Bilingual coverage isn't shown to matter beyond translating prompts. The EN–ZH gaps (Tab. 9) are 0.8–3.0 points and are confounded with category, since the Chinese tasks sit mostly in Code Intelligence. Also, Why is TerminalBench not part of the comparison?

  1. No confidence intervals on a small benchmark. Each task is worth about 1.7 points, yet Tab. 2 ranks models separated by under one point. Tabs. 3 and 5 give no error bars or run counts.
Questions for authors
  1. Does OSWorld qualify as "in-the-wild" by the authos' standards?
  2. What does "native runtime" mean, and what separates check from dimond sign in Tab. 1? What does WildClawBench add on top of existing ones like OSWorld and TerminalBench? Why is TerminalBench not part of the comparison?
  3. hypothetically, what would a Chinese translation of existing OSWorld tasks miss?
  4. Please add confidence intervals to Tab. 2, give run counts and error bars for some key choices of model like Tab 3.
Reviewer_LqfHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WildClawBench, a benchmark for evaluating long-horizon agents operating through native CLI-based harnesses. The benchmark comprises 60 human-authored tasks across six categories, including 26 multimodal tasks and tasks written in English and Chinese. Unlike benchmarks that primarily rely on synthetic tool APIs or final-answer evaluation, WildClawBench executes agents in containerized environments using four established agent harnesses: OpenClaw, Claude Code, Codex, and Hermes Agent.

Critical strengths
  1. Realistic agent evaluation framework. The benchmark addresses an important limitation of existing agent evaluations by executing long-horizon tasks through actual agent harnesses rather than relying exclusively on simplified tool-calling environments.

  2. Solid evaluation and practical findings. The evaluation of 36 frontier models provides a broad empirical comparison. More importantly, the additional experiments demonstrate that performance is substantially affected by the agent harness, execution-time budget, and available skills.

Critical weaknesses
  1. Limited benchmark size and insufficient evidence of task representativeness. Despite evaluating 36 models, the benchmark contains only 60 tasks distributed across six heterogeneous categories, with some categories represented by relatively few examples. More importantly, the task-curation procedure (Section 3.3) explicitly selects tasks based on performance differences between pilot models, potentially favoring tasks that discriminate among those models rather than tasks representative of real-world agent workloads.

  2. Insufficient validation of the hybrid grading methodology. Although the authors introduce a comprehensive grading framework, its reliability is validated on a relatively limited sample. The human–LLM agreement study (Appendix K, Table 13) considers only five tasks, while the judge-sensitivity analysis (Table 4) covers four evaluated models. Although the reported agreement is encouraging, it remains unclear whether the grading procedure is equally reliable across all six task categories, particularly subjective multimodal tasks and safety-related evaluations.

Questions for authors

I appreciate the effort invested in constructing WildClawBench and recognize its potential as a valuable community benchmark. I would be willing to raise my score if the authors could satisfactorily address the following points:

  1. Clarify and justify the task-selection methodology. Provide the number of initially constructed tasks, the number removed during filtering, and the models used in the discriminability-based selection procedure. Explain how the authors ensured that the final 60 tasks remain representative of realistic agent workflows rather than being disproportionately tailored to distinguish the pilot models.

  2. Strengthen the validation of the grading methodology. Extend the human–LLM agreement analysis to a more representative sample of tasks covering all six categories, particularly Creative Synthesis and Safety Alignment.

  3. Better characterize the real-world evaluation setting. Clarify which tools and external services operate through real integrations and which rely on controlled or mocked components.

  4. Provide stronger evidence of ranking stability. Given the relatively small number of heterogeneous tasks, report uncertainty estimates or confidence intervals for aggregate scores and major model comparisons.

Reviewer_Lsx6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a benchmark consisting of 60 native-runtime agent tasks across 6 different categories that, unlike many previous benchmarks, rely on real tools, making the benchmark more realistic. It includes multimodal tasks, two languages, 4 agent harnesses, and evaluates 36 models. The benchmark uses a hybrid approach combining rule-based checks and LLM/VLM judges to grade the models. The results show that the best-performing model, GPT 6 Astra, achieves only 70.1%, with only five other models scoring above 60%, emphasizing that current models are still struggling with these task types.

Critical strengths

They evaluate 36 models across 4 agent harnesses, 6 task categories, 2 languages, and 2 modalities, providing broad experimental coverage. This allows comparisons of agent performance across different settings and highlights the effect of the harness.

Critical weaknesses
  1. Insufficient evidence and questionable filtering: The evidence is not sufficient to support the paper’s claim of being real-world. The tasks are authored by the researchers, and there's not enough evidence that they represent real user requests, and the number of tasks is small (only 60). Also, we don't know which models were used in the pilot phase and whether they overlap with the evaluated models. The filtering is also questionable, as they keep only tasks where at least two pilot models differ by at least 0.2 in score. This makes it unclear whether the benchmark would generalize to other models and whether the findings would still hold if more tasks are added.

  2. Limited evaluation reliability: Although they have two human annotators, the agreement between them is not reported, and the human–LLM judge agreement study is very limited. This makes it unclear to what extent the findings match human judgments and are reliable. Also, some top models differ by only 0.1 to 0.2 points (eg. 67.2 vs. 67.1), so we do not know if the difference is actually meaningful or if it's noise.

Questions for authors
  1. How do the authors justify that the selected tasks are representative of real-world agent workflows?
  2. Given the small score differences between some top models, have the authors checked whether these differences are meaningful?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.