acceptodds
Under review as a conference paper at ICLR 2027

Qualifying AI Agents Under Cost-Aware Human Oversight

Abstract

As AI agents move into real workflows, evaluation must answer more than whether an agent can complete a task. Deployment requires determining under what operating conditions a system can meet a specified reliability requirement at acceptable cost, and whether the available evidence supports that operating point. We introduce Reliable Enterprise Agent Deployment (READY), a framework that turns workflow-level evaluation evidence into a statistically supported deployment qualification. READY preserves each workflow’s definition of successful execution while evaluating the complete agent–workflow–oversight configuration. Given a prespecified class of oversight policies, it evaluates their reliability and operating cost, tests them on held-out cases with finite-sample error control, and returns the lowest-cost member of the certified set. The resulting deployment profile characterizes reliability, autonomous coverage, human-oversight burden, and cost rather than reducing a system to a single benchmark score. We instantiate READY in the terminal accept-or-escalate setting on CliniCARE-Bench clinical audit (18 systems, 750 cases) and τ2-bench retail (17 systems, 114 tasks). Across both workflows, autonomous accuracy shows no clear correlation with how well self-reported confidence ranks successes above failures (Pearson r = −0.07 and −0.13). Consequently, systems with similar autonomous performance can require substantially different oversight to meet the same reliability target. On clinical audit, Sonnet 5 and GPT-5.4 differ by only 0.3 points in development accuracy, yet their certified operating points require 12.0% versus 29.3% human review at the same target. Accounting for agent and review costs can further change which certified configuration is preferred. These results show that deployability is not an intrinsic property of an agent or its benchmark score, but of an agent-workflow- policy configuration under explicit assumptions, with routing quality, statistical uncertainty, and cost jointly shaping the reliability-oversight-cost tradeoff.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, deployment, human oversight, calibration, reliability, cost-aware evaluation, enterprise AI, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iQ3WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work claims that there is a gap in current benchmarks in that they do not account for human review, which would be present in an actual deployed setting. The effectiveness of human review is very dependent on the model's understanding of its own uncertainty, so models that know when they are likely wrong are cheaper to fix with human review (as you have to review fewer cases, only the ones the model flags). Our benchmarks should be adjusted accordingly, giving higher scores to models that are calibrated. And this paper proposes a simulation setup to do that.

Critical strengths
  • Measuring how well model's understand their own certainty is important for deployed settings and should be evaluated in benchmarks
Critical weaknesses
  • This is one of the worst-written papers I have reviewed this year. It is so bad that it feels fully AI written. I could barely understand it. It needs to be rewritten from scratch, by a human, with a focus on clarity. As a simple but clear example of the lack of care put into this work, look at the following in the reproducability statement: " To support independent reproduction, an anonymized version of the reference testbed together with the evaluation configurations, split seeds, and analysis scripts used to produce the reported tables and figures is provided as supplementary material." There is no attached supplement of course. The lack of code release, combined with the poor quality of writing, means that I can't properly review or understand the experimental setup.

  • I think the overall proposal is not good. I agree that benchmarks should also measure internal model uncertainty estimates, as it is useful to know if the model accurately captures when it is right/wrong, but I think the more complex simulation proposed by this work is bad since it adds additional assumptions/complexity. It would be better to simply add brier score or something of that nature to standard benchmark metrics.

Questions for authors

None

Reviewer_CciLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper targets at a famous learning-to-defer question. The authors design a framework to evaluate whether AI can make human-AI team performance achieve a utility target (the threshold), while assuming a budget of human oversight. They offer demonstrations of this framework on two recent benchmarks to evaluate a series of LLM models.

Critical strengths
  • The demonstrations in the paper are attractive, with new benchmarks and many closed-sourced LLM models.
Critical weaknesses
  • The proposed framework does not offer clear distinction between the prior works. This cost-aware delegation problem has been studied in human-AI literature for a long time. It is hard to evaluate the proposed framework without necessary comparison with other baseline methods in the experiment.
  • My second main concern is that the assumption of the human reviewer is not realistic. The authors just assume 90% accuracy of the human (as a prior uniformly on every instance). This strong assumption might make the framework much less useful in practice. For example, people and AI could be good at different instances, so delegating to human reviewers's performance is not equivalent to one accuracy number.
Questions for authors
  • How would it like to validate the framework and its generated benchmarks on LLM models in a practical task? Could the framework be generalized to actual human reviewers (instead of just an accuracy).
  • What is the difference between their framework with other works in risk control such as [1]? I also see there is another work published on this NeurIPS [2] working on the similar problem.

[1] Laufer-Goldshtein, Bracha, Adam Fisch, Regina Barzilay, and Tommi Jaakkola. "Efficiently controlling multiple risks with pareto testing." arXiv preprint arXiv:2210.07913 (2022).

[2] Pona, Edoardo, Milad Kazemi, Mehran Hosseini, Yali Du, David Watson, Osvaldo Simeone, and Nicola Paoletti. "Calibrate-Then-Delegate: Safety Monitoring with Risk and Budget Guarantees via Model Cascades." arXiv preprint arXiv:2604.14251 (2026).

Reviewer_7kGtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces READY, a framework for deciding how much human oversight an AI agent needs to meet a reliability target at minimum cost. It uses held-out statistical tests to qualify oversight policies and evaluates the approach on clinical audit and retail tasks, revealing agents with similar accuracy can require very different amounts of human review.

Critical strengths
  1. The paper asks a practical question that current AI agent benchmarks often misses. Comparing systems by the human effort needed to achieve a given reliability target is more informative than comparing accuracy alone.

  2. The experiments cover a good range of models, and the authors provide useful sensitivity analyses. The held-out testing procedure and the comparison with naive threshold selection are particularly convincing.

Critical weaknesses
  1. The main results assume that human review succeeds 90% of the time, regardless of case difficulty. No actual human review is conducted. Since reviewing every case meets the 75% reliability target by assumption, the reported qualification results tell us more about the routing policies than about whether these systems can reliably be deployed.

  2. The framework builds on established selective prediction and Learn-then-Test methods. Within each system, agent cost is fixed, so minimizing cost simply amounts to minimizing the review rate.

Questions for authors
  1. How would the qualification results change if human-review success depended on case difficulty? Can the authors provide evidence supporting the assumed 90% success rate?

  2. How does READY compare with a standard risk-controlled selective prediction or deferral baseline using the same data and policy family?

  3. In introduction, "AI agents increasingly perform ..." sounds AI-generated/patterned.

  4. In related works for Routing signals, human oversight, and assurance, TeamBench (2026) can be mentioned.

  5. In Table1, opencode can be OpenCode?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviews

Dear Area Chair and Reviewers,

Thank you for your time and careful feedback on our submission. We agree that the main concerns need more work than the discussion period allows.

In a future version, we will let human-review accuracy vary with case difficulty and base it on real human-review data. We will also compare READY directly with risk-controlled selective prediction and learning-to-defer methods on the same data, including Pareto Testing and Calibrate-Then-Delegate, and explain more clearly how READY differs. Finally, we will rewrite the paper for clarity and add the related work the reviewers suggested.

We will release the code and evaluation configurations with the revised version.

Thank you again for your helpful comments.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.