- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper argues for small-sample, long-running agent evaluations with qualitative log analysis as a complement to benchmarks. It proposes five dimensions for describing these evaluations, surveys existing examples, and introduces CRUX. Its first study gives an OpenClaw agent using Claude Opus 4.6 the task of building and publishing a simple iOS app. The app was published after five manual interventions, one classified as avoidable. Log inspection identifies a fabricated contact number, output defects, and expensive polling during the review wait. The paper derives recommendations on reporting constructs, interventions, logs, monitoring, dry runs, and costs.
Critical strengths
-
The case study records findings that publication success alone would conceal. The fabricated phone number, ineffective sound toggle, and polling cost are concrete examples of why an evaluator should inspect the execution and artifact. Reporting these failures makes the case more useful than a success demonstration.
-
The proposed reporting practices address a real problem with interpreting public agent demonstrations. Separating setup effort, human intervention, and agent cost would help readers compare the evidence across studies. Section 2.4 also acknowledges the limitations of small samples and changing environments rather than presenting CRUX as a model-ranking benchmark.
Critical weaknesses
-
The conceptual novelty is limited. The studies surveyed in the paper already use real-world tasks, extended execution, human intervention, and qualitative log analysis. Grouping these practices under the term "open-world evaluations" provides a useful overview, but the paper does not clearly establish what the proposed concept adds to those existing approaches. The five dimensions describe a broad range of studies, with flexible boundaries and exceptions. It remains unclear how this classification changes task selection, intervention decisions, or the interpretation of results in a way that existing evaluation practices do not.
-
The research contribution beyond the survey and deployment case is unclear. The six recommendations, including documenting interventions, releasing logs, and reporting costs, are sensible research practices, but they are not developed into a concrete, reusable evaluation protocol. For example, CRUX does not specify rules that another evaluator could apply to classify human assistance or systematically identify and assess events in the logs. The iOS experiment provides interesting observations, particularly the fabricated phone number and polling costs, but it does not establish the added methodological value of the proposed framework
Questions for authors
I have no questions for authors.
Response to Reviewer aQmm
Thank you for the detailed review.
Contribution relative to earlier evaluations
We agree that prior work has used some of these aspects of evaluations. For example, the C compiler study tells us that the compiler could build Linux, and the authors themselves went to some trouble to describe where it still fell short for a developer who wanted to use it. What we think is missing is a procedure or a methodology that can provide guidance for open-world evaluations to become another complementary aspect of evaluations alongside benchmarking.
Task selection and the five dimensions
In CRUX #1 we deliberately allowed the agent to choose a simple app. It chose a timer for breathing exercises. If we'd wanted to know whether the agent could code a more complicated iOS app, there are quite good ways to test that already and little reason to involve Apple. What we wanted to know was whether a developer could delegate getting an app out onto the actual App Store. E.g., suppose we had mocked the signing service so we could reset the environment a hundred times. One consequence would have been to omit precisely the macOS dialog that the agent had difficulty with in our run. It tried to click the dialog from software, an approach it had successfully used elsewhere in the GUI. We chose to use a real Mac and developer account, and to keep the run going through human review and release. We stopped both dry runs before submission so that Apple's reviewers would not have to spend time reviewing apps we were using to test our setup. Had we wanted the expected success rate for coding an app, we could instead have specified 50 app ideas in advance, run the agent on all of them and skipped the App Store entirely. Starting from much the same question about whether agents can build apps, we would then have ended up with a different environment, task duration, and sample size.
Classifying human assistance
We would start with a decision that has to precede the experiment. If one of us clicks Allow on a protected dialog, can the run still count as a success? In our case the answer was yes, since obtaining the operator's authorization was part of the task. Another group could reasonably count that same click against success, e.g., if the question they wanted to answer was whether the system could run overnight without anybody there to operate the Mac. In either case we'd keep a record of what prompted the intervention, what information or action the human supplied, the time it happened, and what the agent did subsequently. Searching the agent's messages for requests for help would not turn up the occasion on which we had to restart OpenClaw.
The labels we used in this case were a platform requirement (a physical click in the example above), a restriction we had imposed (get our permission before public release), a problem with our own setup (the OpenClaw crash), or a step the agent could have completed with what it already had (using its credentials after the browser session expired).
Verifying findings and releasing the protocol
For the analysis itself we used Docent to search the agent traces, summarize the agent's actions, and find the relevant tool calls. One of the authors checked each claim against the messages, tool calls, and screenshots.
We will work through this example in Sec. 4, starting with the task and release condition, and include the files we used to run it with the camera-ready. We couldn't point to all of the artifacts during anonymous review, since the app, its repo, and the site hosting the logs identify the authors. We will release the prompts and setup, the scrubbed logs with the intervention records and worked annotations, the cost calculation, and instructions for conducting a new experiment alongside the camera-ready version.
It is an open question whether a team running a different study would find the same kinds of failures, particularly for a task where it isn't as straightforward to inspect the result as it was to download our app. We hope other groups will use the package to investigate that question.