acceptodds
Under review as a conference paper at ICLR 2027

AUTHBENCH: Benchmarking Interactive User Authorization in Computer-Use Agents

Abstract

Computer-use agents act on their users' behalf, and delegation is legitimate only if every consequential action stays within what the user permitted. However, existing benchmarks ask only whether the agent reached the requested state, and those that score constraints fix them in advance, so nothing said during a run can grant or withdraw a permission. We argue that authorization is instead an in-run grant: a permission the user issues while the agent works, bound to the values they were shown and void once those values change. Grants are private, perishable, and invisible in the final state. We realize this in AUTHBENCH, an interactive environment over complete web applications, where a simulated user holds authorization privately, state can change underneath an approved action, and an append-only trace records every request and attempted action. On it we build a benchmark of authorization-sensitive tasks with executable oracles and evaluate three deployed computer-use systems. Our results show that capability and authorization come apart: the system that completes the most tasks stays within its user's grants least often, none is both successful and authorized in even half its runs, and not one confirmation was bound to the action it authorized. AUTHBENCH thus makes authorization measurable as a property distinct from capability, and we release the environment, the tasks, and every recorded run.

Primary area:
datasets and benchmarks
Keywords:
authorization, compliance, benchmark, llm agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gfKoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AuthBench, a new computer use benchmark for measuring how well computer use agents conform to user specified authorization. AuthBench consists of authorization sensitive tasks, with grants issued by simulated users. Agents are measured on their ability to complete the tasks while respecting the user granted permissions. They find that agents which achieve stronger success rates typically violate permission grants more frequently (and vice versa).

Critical strengths
  • The use case of benchmarking CUAs for how well they respect user permissions is a very timely and well motivated benchmark. Most existing CUA evals focus on capabilities or safety, but respecting user permissions is a nice in-between and very well motivated for current product use cases of CUAs.
  • Tasks come from public issue reports that describe a failure an agent or its authorization mechanism commits, which make them realistic and grounded in real world use cases.
  • The baselines run are reasonable, though I'd encourage the authors to try Opus 5.5 and gpt-6-astra models as well (which are step improvements over the models run here on CUA evals).
Critical weaknesses
  • There is limited analysis and ablations over the baselines. In particular, I'd like to see what interventions could potentially improve the compliance of the agents. Common safeguards, like safety filters (prompting a smaller model to classify actions as safe or not), or prompt-based interventions would be most useful to see as ablations.
  • All baselines are proprietary API based models, what about open weight models? In particular, Qwen3.8-27B and Kimi-K3 are close-to-SOTA on CUA for their size. Having some baseline results on open weight models would be essential to facilitate the reproducibility of future work.
  • Baselines seem to be mostly based on the responses API, it would be nice to see some models run through their harnesses (e.g., running codex-cli with the computer use tools) which may include other safeguards and optimizations not present in the API-based implementations.
Questions for authors
  • Can you show results of more baselines, including recent frontier models (Opus 5.5, gpt-6 family) and some open weight models (Qwen3.8-27B, Kimi-K3)?
  • Why are the OpenAI models so sensitive to persona changes?
  • How important or relevant is computer use for AuthBench? How many of the tasks can be done through alternative means, e.g. using bash or web search? How do agents fare if they use these text-based interfaces instead? I could imagine models being more robust if they interact through these tools, rather than using full computer use.
Reviewer_3UotOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that user authorization for computer-use agents should be modeled as an in-run grant: a permission the user issues mid-task, bound to the field values shown at confirmation (recipient, amount, fee, role), single-use, and void once any covered value changes. From this it derives three requirements (interactive acquisition, stateful binding, auditable traces) and builds AUTHBENCH to meet them: 268 tasks over five locally hosted cloned web applications (email, payments, cloud storage, shopping, team chat), an LLM-simulated user (Claude Sonnet 4.6, five personas) that holds authorization privately, a controller that can rewrite approved values after confirmation, and a deterministic trace validator computing AS=T⋅F⋅AAS = T \cdot F \cdot A (goal reached, no invariant broken, every consequential action authorized at commit time).

Three deployed systems (Claude, Gemini, OpenAI computer-use) are each run on 268×5=1,340268 \times 5 = 1{,}340 runs. Under the strict policy, authorization compliance is 25.9 / 48.2 / 55.9%; under a relaxed policy that forgives missing confirmations it becomes 89.6 / 61.7 / 80.9%. No confirmation from any system is scored as bound to the exact action it authorized.

Critical strengths

The paper captures a real problem that no one has systematically measured before. Existing benchmarks look only at task outcomes, not at whether the agent was entitled to act. This paper is the first to formalize "approval covers only what the user saw, and lapses once a value changes" and turn it into an executable benchmark. Matched pairs change one factor at a time, so intervention effects admit a causal reading.

Why it matters: this is the main reason I do not recommend rejection. Even without the cross-system comparison, the formulation and environment are valuable on their own.

Critical weaknesses

W1. The headline conclusion is mainly determined by the scoring policy and does not directly show behavioral differences between systems.

The claim that "the system that completes the most tasks is the least compliant" holds only under the strict policy. Under the relaxed policy, Claude moves from the lowest compliance (25.9%) to the highest (89.6%), because most of its violations are simply not asking the user, not doing something the user explicitly disallowed. In addition, Appendix I shows that 370 Gemini runs (about 28% of its scored runs, by my calculation from Table 3) were counted as failures because the product cannot ask the user a question; the main text does not mention this handling.

Why it matters: the core claim is that "task capability and authorization compliance come apart," but this depends heavily on which scoring policy is used. It can be fixed by rewording and reporting results under the other policies, which is why I give weak accept rather than weak reject.

W2. The credibility of the results lacks necessary validation.

The paper says it reports oracle error rates and simulator agreement, but these numbers cannot be found in the appendices. The authors also acknowledge that agreement between the simulator and human judgment has not been verified. Every grant is issued by a user simulated with Claude Sonnet, while one evaluated system is Claude Opus; the two belong to the same model family, and the authors did not check for bias. The paper also does not compare results between the cloned applications and real products, and provides no human baseline.

Why it matters: compliance rates depend directly on the simulator's decisions. Without even a small human-agreement study, one cannot tell whether differences in compliance come from the evaluated systems or from the simulator. Such a study is modest in effort and feasible during the rebuttal.

Questions for authors

Q1 (W1). Please report the effect of Gemini's 370 runs on Table 3 under both the "exclude" and "count as failure" policies, and state this in the main text.

Q2 (W2). Could you add a small human study (e.g., 3 annotators labeling 100 confirmation requests) and report agreement between humans and the simulator (e.g., Cohen's κ\kappa)?

Q3 (W2). Could you run a small batch of neutral tasks in a real product's official test environment to verify that results from the cloned environment transfer? Also, could you report a human operator's authorized success rate under the strict policy?

Reviewer_SyQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AUTHBENCH, a benchmark for evaluating whether computer-use agents perform consequential actions within user-authorized boundaries. It models authorization as interactive, state-dependent grants and combines LLM-simulated users, controlled environment interventions, and deterministic trace-based oracles. Across 268 tasks in five web applications, the evaluation of three deployed systems reveals a substantial gap between task success and authorization compliance. Claude achieves 94.5% task success but only 25.9% authorization compliance, while Gemini achieves highest authorized success rate ( 42.5%).

Critical strengths
  1. A precise, operational definition of authorization that the trace makes checkable. AUTHBENCH explicitly evaluates dynamically acquired authorization, including permission scope, revocation, and invalidation after state changes. AUTHBENCH introduces deterministic executable oracles that evaluate authorization throughout the agent's execution trajectory.
Critical weaknesses
  1. Potential misalignment in authorization ground truth. The benchmark relies on author-defined tasks and authorization policies whose alignment with real users' intentions has not been independently validated. Moreover, agents are evaluated against authorization requirements that may be implicit or hidden from their instructions, making it unclear whether violations reflect genuine failures to respect user permissions or reasonable interpretations of underspecified requests.

  2. Confounding between agent behavior and authorization mechanisms. The evaluated systems have different native confirmation capabilities, yet the benchmark compares them using aggregate authorization compliance. Claude's compliance increases from 25.9% to 89.6% under relaxed scoring, suggesting that much of the measured difference reflects missing confirmation mechanisms rather than violations of explicit user constraints

Questions for authors

See weaknesses, and:

  • Does the agent receive a fresh observation after an environmental intervention and before execution? Clarifying this would help distinguish failures to recognize stale authorization from limitations of the confirmation mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.