acceptodds
Under review as a conference paper at ICLR 2027

Open-World Evaluations for Measuring Frontier AI Capabilities

Abstract

Benchmark-based evaluation remains important for tracking frontier AI progress. But it can both overstate and understate deployed capability because it privileges tasks that can be precisely specified, automatically graded, easy to optimize for, and run with low budgets and short time horizons. We advocate for a complementary class of evaluations, which we term open-world evaluations: long-horizon, messy, real-world tasks assessed through small-sample qualitative analysis rather than benchmark-scale automation. In this paper we survey recent open-world evaluations, identify their strengths and limitations, and introduce CRUX (Collaborative Research for Updating AI eXpectations), a project for conducting such evaluations regularly. As a first instance, we task an AI agent with developing and publishing a simple iOS application to the Apple App Store. The agent completed the task with only a single avoidable manual intervention, suggesting that open-world evaluations can provide early warning of capabilities that may soon become widespread. We conclude with recommendations for designing and reporting open-world evals.

Primary area:
datasets and benchmarks
Keywords:
open-world evaluations, frontier AI capabilities, benchmark validity, LLM agents, long-horizon tasks, computer-use agents, log analysis, evaluation, AI policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aQmmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues for small-sample, long-running agent evaluations with qualitative log analysis as a complement to benchmarks. It proposes five dimensions for describing these evaluations, surveys existing examples, and introduces CRUX. Its first study gives an OpenClaw agent using Claude Opus 4.6 the task of building and publishing a simple iOS app. The app was published after five manual interventions, one classified as avoidable. Log inspection identifies a fabricated contact number, output defects, and expensive polling during the review wait. The paper derives recommendations on reporting constructs, interventions, logs, monitoring, dry runs, and costs.

Critical strengths
  1. The case study records findings that publication success alone would conceal. The fabricated phone number, ineffective sound toggle, and polling cost are concrete examples of why an evaluator should inspect the execution and artifact. Reporting these failures makes the case more useful than a success demonstration.

  2. The proposed reporting practices address a real problem with interpreting public agent demonstrations. Separating setup effort, human intervention, and agent cost would help readers compare the evidence across studies. Section 2.4 also acknowledges the limitations of small samples and changing environments rather than presenting CRUX as a model-ranking benchmark.

Critical weaknesses
  1. The conceptual novelty is limited. The studies surveyed in the paper already use real-world tasks, extended execution, human intervention, and qualitative log analysis. Grouping these practices under the term "open-world evaluations" provides a useful overview, but the paper does not clearly establish what the proposed concept adds to those existing approaches. The five dimensions describe a broad range of studies, with flexible boundaries and exceptions. It remains unclear how this classification changes task selection, intervention decisions, or the interpretation of results in a way that existing evaluation practices do not.

  2. The research contribution beyond the survey and deployment case is unclear. The six recommendations, including documenting interventions, releasing logs, and reporting costs, are sensible research practices, but they are not developed into a concrete, reusable evaluation protocol. For example, CRUX does not specify rules that another evaluator could apply to classify human assistance or systematically identify and assess events in the logs. The iOS experiment provides interesting observations, particularly the fabricated phone number and polling costs, but it does not establish the added methodological value of the proposed framework

Questions for authors

I have no questions for authors.

1 comment
AuthorsOpenReview ↗

Response to Reviewer aQmm

Thank you for the detailed review.

Contribution relative to earlier evaluations

We agree that prior work has used some of these aspects of evaluations. For example, the C compiler study tells us that the compiler could build Linux, and the authors themselves went to some trouble to describe where it still fell short for a developer who wanted to use it. What we think is missing is a procedure or a methodology that can provide guidance for open-world evaluations to become another complementary aspect of evaluations alongside benchmarking.

Task selection and the five dimensions

In CRUX #1 we deliberately allowed the agent to choose a simple app. It chose a timer for breathing exercises. If we'd wanted to know whether the agent could code a more complicated iOS app, there are quite good ways to test that already and little reason to involve Apple. What we wanted to know was whether a developer could delegate getting an app out onto the actual App Store. E.g., suppose we had mocked the signing service so we could reset the environment a hundred times. One consequence would have been to omit precisely the macOS dialog that the agent had difficulty with in our run. It tried to click the dialog from software, an approach it had successfully used elsewhere in the GUI. We chose to use a real Mac and developer account, and to keep the run going through human review and release. We stopped both dry runs before submission so that Apple's reviewers would not have to spend time reviewing apps we were using to test our setup. Had we wanted the expected success rate for coding an app, we could instead have specified 50 app ideas in advance, run the agent on all of them and skipped the App Store entirely. Starting from much the same question about whether agents can build apps, we would then have ended up with a different environment, task duration, and sample size.

Classifying human assistance

We would start with a decision that has to precede the experiment. If one of us clicks Allow on a protected dialog, can the run still count as a success? In our case the answer was yes, since obtaining the operator's authorization was part of the task. Another group could reasonably count that same click against success, e.g., if the question they wanted to answer was whether the system could run overnight without anybody there to operate the Mac. In either case we'd keep a record of what prompted the intervention, what information or action the human supplied, the time it happened, and what the agent did subsequently. Searching the agent's messages for requests for help would not turn up the occasion on which we had to restart OpenClaw.

The labels we used in this case were a platform requirement (a physical click in the example above), a restriction we had imposed (get our permission before public release), a problem with our own setup (the OpenClaw crash), or a step the agent could have completed with what it already had (using its credentials after the browser session expired).

Verifying findings and releasing the protocol

For the analysis itself we used Docent to search the agent traces, summarize the agent's actions, and find the relevant tool calls. One of the authors checked each claim against the messages, tool calls, and screenshots.

We will work through this example in Sec. 4, starting with the task and release condition, and include the files we used to run it with the camera-ready. We couldn't point to all of the artifacts during anonymous review, since the app, its repo, and the site hosting the logs identify the authors. We will release the prompts and setup, the scrubbed logs with the intervention records and worked annotations, the cost calculation, and instructions for conducting a new experiment alongside the camera-ready version.

It is an open question whether a team running a different study would find the same kinds of failures, particularly for a task where it isn't as straightforward to inspect the result as it was to download our app. We hope other groups will use the package to investigate that question.

Reviewer_d6kaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new style of evals for the near future - where we don't constrain our benchmarks to only include tasks that humans can solve and where we have ground truth (or oracle) solutions. The paper clearly communicates the risks of using other agentic evals.

Critical strengths

The paper is ahead of its time in suggesting and recommending best practices for modern AI evals. It proposes a perspective and framework for building evals that are more open-ended and realistic. The criteria outlined in this paper are a useful way to think about these evals and are novel - especially criteria like counting and reporting the number of human interventions. The paper clearly documents the cost of running such an eval and clearly communicates why this is hard.

Critical weaknesses

This paper is based on one evaluation task and we don't have enough runs/task diversity to draw clear conclusions. The paper also doesn't share the logs or data for this experiment - making reproduction and verification harder.

Questions for authors

Would the authors consider sharing more information/data about the task, the related logs and task success/failure information? It would also help to include a few more tasks or runs - since this paper is based on very little evidence.

1 comment
AuthorsOpenReview ↗

Response to Reviewer d6ka

Thank you for the review.

Availability of the data

Yes, we are happy to share the data. Much of the material you ask for is already available, but linking to it (including the iOS app itself) would identify the authors. We will make the data and logs available alongside the camera-ready. In particular, we'll include both the files that the agent saw when we ran the experiment and the changes we made to the setup after the first and second dry runs. It took us roughly eight person-hours just to get the Mac VM, developer accounts, and logging working with OpenClaw. We would like to save other researchers the work of reconstructing that setup. We will include the code for reproducing the cost analysis and a description of how to use the same setup to run one's own experiments.

Evidence for success and failure

With respect to the task itself, the instruction we gave the agent at launch was “Read AGENTS.md and get started.” We put the task description in USER.md. We asked for an app that someone would actually want to download, leaving the choice of what kind of app to build up to the agent. The choice of a breathing-exercise app was the agent's. Apple approved the submission, but some of the evidence about how the agent completed the task was only apparent from the trace. For example, we had told the agent to ask us for a phone number when it needed one for the review contact, and it had in fact asked for our help with other credentials. For the phone number, it instead tried several values and included a number reserved for fictional use in the form it sent to Apple. We were able to reconstruct this part of the submission from the agent's tool calls. We will annotate this event along with the five human interventions in the release, since a reader looking only for requests for help could quite reasonably miss it. The agent never asked us for a number.

Repeated runs and the scope of the result

We discussed repeated runs when planning the iOS evaluation, for much the same reason you raise in the review. We felt it was reasonable to submit one AI-generated app, but were reluctant to submit dozens, since each submission asks Apple's reviewers to do work for our experiment. We did run the experiment twice before the actual submission to make sure we had the setup right, but stopped both of those runs before submission for this reason. We don't intend to make a claim about more complex apps, changed platform procedures, or what would have happened had the app undergone a substantive negative review by Apple.

Reviewer_PyGsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a small number of expensive, realistic, long-duration tasks, combined with intervention records and detailed log analysis, can complement standardized benchmarks. It characterizes a flexible family of open-world evaluations, surveys illustrative cases, and reports an app-development and App Store publication case using Claude Opus 4.6 with OpenClaw. Its reporting recommendations emphasize the target construct, human intervention, logs, monitoring, dry runs, and costs. The case demonstrates conditional feasibility while exposing operational, quality, and integrity failures that a binary success label would miss.

Critical strengths
  1. The case makes process evidence consequential to interpreting success. Credential recall failure, a fabricated review phone number, expensive polling, and a nonfunctional sound switch reveal different limitations despite publication success. Including these failures, five actual interventions, and setup costs makes the case informative about what an outcome-only evaluation omits.

  2. The paper offers useful guidance on defining and reporting the measured capability. The app case primarily tests navigating a real publication workflow rather than sophisticated software engineering. Explicitly separating that construct from product quality, and reporting human involvement and cost, provides a useful methodological contribution for evaluating deployed agent systems.

Critical weaknesses
  1. The empirical support is an existence demonstration, not an estimate of reliability or a statistical capability upper bound. One successful run, preceded by two dry runs and substantial setup, cannot establish typical success, robustness, or the cost distribution. Much of the roughly ten-day duration is platform waiting rather than active complex planning. Five actual interventions, of which one is classified as avoidable, must remain distinct from a claim of only one intervention. These limitations materially restrict the capability conclusions even though the paper acknowledges several of them.

  2. The proposed framework needs a more reproducible operational protocol. Flexible category boundaries, subjective intervention classification, and qualitative log analysis make cross-team comparisons difficult. Whether infrastructure failures count depends on whether the construct is model capability or full-system reliability. The endpoint also needs precision: platform approval, public release, and a functional product are different outcomes. Finally, the statement that full logs have been released (p. 9) and the reproducibility statement that they are planned for release (p. 10) leave independent auditability unclear.

Questions for authors
  1. Please enumerate all five interventions with triggers, actions, time spent, and information supplied. What exactly was communicated in the credential reminder, and how would alternative reasonable classifications change the interpretation?
  2. Was the measured endpoint approval or public release, and what human action was needed for release? Please reconcile the March 14 approval and March 16 release markers with the textual timing and separate active execution, platform waiting, and total elapsed time.
  3. Are the complete sanitized logs currently accessible? Please resolve the conflicting release statements and identify the evidence for all interventions, the fabricated phone number, and cost changes. Which configuration and prompt changes resulted from the two dry runs?
  4. Please distinguish setup, development, polling, and intervention costs, including the change from 30-minute to 5-minute wakeups. Were independent functional, security, or privacy checks performed beyond platform approval? Repeated runs, task variations, and handling substantive review rejection would be valuable future evidence, but are not established by this single successful case.
1 comment
AuthorsOpenReview ↗

Response to Reviewer PyGs

Thank you for the review. We'll release the workspace files and OpenClaw setup with the camera-ready, along with instructions for running other experiments.

Human interventions (UTC)

  1. Apple ID password. On March 6 at 21:25 the agent needed to sign the SwiftUI app in Xcode. We had kept the Apple ID password out of its workspace. We sent the password in chat and the agent entered it in Xcode.
  2. Two-factor authentication. At 21:38 on March 6 the agent asked us to click Allow on the macOS dialog, after trying unsuccessfully to do this from software. One of us clicked Allow for it (six minutes after the request) and re-enabled the screenshot script. The agent read and entered the code itself.
  3. Credential reminder. On March 13 at 17:02 the agent asked for help because its browser session for checking review status had expired. The complete human message is “Go sign in with the password we gave you before, resolve the 2FA yourself if you can”. Six minutes later it reported finding the App Store Connect API key in its workspace and used that to resume checking the review status. Our reply referred to the password it already had. We did not tell it about the key or give it the path.
  4. Daemon restart (March 16). OpenClaw had crashed so we restarted the daemon, with logging resuming at 16:09. The operator's spreadsheet records the restart, which one would miss by looking only for requests for help from the agent.
  5. Public release (March 16, 16:29). The approved app was pending developer release, for which we had instructed the agent to seek our authorization. We gave the go-ahead (“Make sure the app is live on the App Store and can be downloaded”), and a minute later, after calling the release API, the agent observed READY_FOR_SALE.

We classified intervention 3 as avoidable because, when the agent asked us for help, it already had the credentials it subsequently used. We agree with your suggestion to put the five interventions in the abstract and explain there what we mean by avoidable, and plan to clarify this in the camera-ready version.

Endpoint and elapsed time

We asked for an App Store link from which a user could download the app. Meeting that condition required a developer release after Apple's approval, hence intervention 5 above. Development and submission were on the first day. Most of the roughly ten days in Fig. 3 were spent waiting to hear back from Apple, albeit with frequent status checks. We agree that the elapsed time is a poor measure of how long the agent could sustain complex planning.

Logs, dry runs, and anonymity

The scrubbed agent logs are publicly accessible. We withheld the link since one can easily identify the authors from the site (or from the App Store listing itself). We will clarify on p. 10 that what remains to be released with the paper is the reproduction package.

On the first dry run we found a placeholder privacy-policy URL, missing macOS screen-recording and accessibility permissions, and a problem with our Xcode configuration. On the second try the GUI and the hosted privacy policy both worked. We took care to stop both dry runs before submission so Apple wouldn't have to review apps we were using to debug the setup. In reviewing the second dry run we also found that a name left in an input file from the first run was influencing the choice of app, so we removed it. After the dry runs we added command-line GitHub access and told the agent the app should be useful to someone who downloaded it.

Costs and product checks

Setup, including both dry runs, took eight person-hours and about USD 50 in API costs. Fig. 3 reports USD 991 in API costs for the main run and about USD 25 for the initial build. We changed the heartbeat from OpenClaw's default of 30 minutes to 5 minutes. This choice accounted for much of the cost reported in Figure 3, even though the agent reduced its hourly spend from USD 35/hour to USD 3/hour by delegating checks to subagents and moving to shorter memory files. One check every 30 minutes would mean about one-sixth as many checks. If the cost per check had stayed the same we'd have spent roughly USD 190. We did not run the 30-minute configuration. We'll include the usage records and the code for the figure so that others can use their own assumptions.

We tried the dry-run builds via TestFlight and inspected the published app. The sound toggle didn't work and the App Store screenshot had a formatting error. We didn't conduct an independent functional, security, or privacy audit beyond Apple's review. We don't intend to make a claim about more complex apps, changed platform procedures, or what would have happened had the app undergone a substantive negative review by Apple.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.