- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces WildClawBench: 60 human-written, long-horizon agent tasks across six categories. The tasks are bilingual and partly multimodal. Each one runs in Docker under a real CLI harness (e.g., OpenClaw, Claude Code) with real tools, and is graded with rule checks, side-effect audits and an LLM judge. Across 36 models, the best scores 70.1%. Further studies show that harness choice (up to 18 points), time budget and added skills all shift performance substantially. Validation includes three runs per model, a judge-swap study and a small human-agreement check.
Critical strengths
- It evaluates the deployed system, widely. Tasks run the real CLI harness in Docker with real tools, and grading assets are mounted only after the agent exits. This is what makes the harness study (Tab. 3) possible: the same model shifts 10–18 points across scaffolds. The time- and thinking-budget results add to this. It's a useful message for the community and my main reason for supporting acceptance.
Critical weaknesses
- The key differentiating factor of the dataset is not clear, despite using differentiating label like "native runtime" and "bilingual".
The paper calls OSWorld a synthetic sandbox, but OSWorld runs real apps in real OS VMs. In Tab. 1, Claw-Eval differs from WildClawBench only on "native runtime" (⋄ vs. ✓). WildClawBench's Social tasks also use mocked participants and mock Slack/email tools. Bilingual coverage isn't shown to matter beyond translating prompts. The EN–ZH gaps (Tab. 9) are 0.8–3.0 points and are confounded with category, since the Chinese tasks sit mostly in Code Intelligence. Also, Why is TerminalBench not part of the comparison?
- No confidence intervals on a small benchmark. Each task is worth about 1.7 points, yet Tab. 2 ranks models separated by under one point. Tabs. 3 and 5 give no error bars or run counts.
Questions for authors
- Does OSWorld qualify as "in-the-wild" by the authos' standards?
- What does "native runtime" mean, and what separates check from dimond sign in Tab. 1? What does WildClawBench add on top of existing ones like OSWorld and TerminalBench? Why is TerminalBench not part of the comparison?
- hypothetically, what would a Chinese translation of existing OSWorld tasks miss?
- Please add confidence intervals to Tab. 2, give run counts and error bars for some key choices of model like Tab 3.