- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AuthBench, a new computer use benchmark for measuring how well computer use agents conform to user specified authorization. AuthBench consists of authorization sensitive tasks, with grants issued by simulated users. Agents are measured on their ability to complete the tasks while respecting the user granted permissions. They find that agents which achieve stronger success rates typically violate permission grants more frequently (and vice versa).
Critical strengths
- The use case of benchmarking CUAs for how well they respect user permissions is a very timely and well motivated benchmark. Most existing CUA evals focus on capabilities or safety, but respecting user permissions is a nice in-between and very well motivated for current product use cases of CUAs.
- Tasks come from public issue reports that describe a failure an agent or its authorization mechanism commits, which make them realistic and grounded in real world use cases.
- The baselines run are reasonable, though I'd encourage the authors to try Opus 5.5 and gpt-6-astra models as well (which are step improvements over the models run here on CUA evals).
Critical weaknesses
- There is limited analysis and ablations over the baselines. In particular, I'd like to see what interventions could potentially improve the compliance of the agents. Common safeguards, like safety filters (prompting a smaller model to classify actions as safe or not), or prompt-based interventions would be most useful to see as ablations.
- All baselines are proprietary API based models, what about open weight models? In particular, Qwen3.8-27B and Kimi-K3 are close-to-SOTA on CUA for their size. Having some baseline results on open weight models would be essential to facilitate the reproducibility of future work.
- Baselines seem to be mostly based on the responses API, it would be nice to see some models run through their harnesses (e.g., running codex-cli with the computer use tools) which may include other safeguards and optimizations not present in the API-based implementations.
Questions for authors
- Can you show results of more baselines, including recent frontier models (Opus 5.5, gpt-6 family) and some open weight models (Qwen3.8-27B, Kimi-K3)?
- Why are the OpenAI models so sensitive to persona changes?
- How important or relevant is computer use for AuthBench? How many of the tasks can be done through alternative means, e.g. using bash or web search? How do agents fare if they use these text-based interfaces instead? I could imagine models being more robust if they interact through these tools, rather than using full computer use.