acceptodds
Under review as a conference paper at ICLR 2027

AccessWorld: A Benchmark for Computer Use Agents Operating Assistive Technologies

Abstract

Computer use agents typically perceive graphical interfaces through screenshots or accessibility trees and act through mouse clicks and keystrokes. Many people with disabilities instead operate the same interfaces through assistive technologies (ATs), such as screen readers, voice control, and screen magnification. As a result, the action and observation spaces available to a user can differ greatly across interaction modalities. Assistive agents that can operate these technologies could collaborate with the people who rely on them, audit whether software actually works with ATs, and guide users through interfaces that are hard to navigate. There is also no reason to assume a GUI built for sighted mouse users is the right interface for an agent. Yet there are currently no tools or benchmarks for building and evaluating agents that operate ATs. We present AccessWorld, a benchmark of computer use tasks on macOS that agents complete by using system native assistive technology. Agents operate on the environment through an OpenClaw harness exposing each AT as an agent tool interface. We evaluate five models across five interaction modalities and find substantial differences across ATs in both task success and interaction cost. Through VoiceOver, macOS's screen reader, one model nearly matches its standard computer-use success, but requires substantially more actions and cumulative input. At magnification, three of five models exceed their standard computer-use accuracy, while increasing magnification to reduces accuracy for all five models. Voice Control produces the lowest accuracy for every model, with the largest drops on multi-item, spreadsheet, and text-editing tasks. AccessWorld measures how well agents handle the constrained observation and action spaces that AT users work with every day.

Primary area:
datasets and benchmarks
Keywords:
computer use agents, accessibility, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M3N2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AccessWorld, a benchmark of 124 macOS tasks, uses existed questions but changes the tools to VoiceOver, Zoom and Voice Control. Five models are tested under five interaction settings. The main finding is that some assistive settings reach or even pass the success rate of normal interaction, but they need much more interaction to do so. Another finding is that the ranking of models under normal interaction does not carry over to the assistive settings.

Critical strengths

The evaluation is carefully executed and goes beyond headline accuracy: the paper characterises how each assistive-technology condition fails — budget exhaustion under Voice Control, fixed-magnification behaviour under Zoom, and false completion claims in self-reporting — and turns each into a concrete design implication. This diagnostic value is what makes the contribution consequential despite the inherited task pool.

Critical weaknesses

Tasks are only checked to be solvable in the reference environment, not under each assistive setting. It is not interesting that limiting the tools for an agent harms the scores. But we dont need limit the agent. "Phone use" usually uses assistive tools, and I dont think macOS is special. Moreover, "Phone use" uses assistive tools because it is simpler than vision, but PC agent has more paths to control.

Questions for authors
  1. Is the limited tools harms the informations? i.e. does it change the solvability of the questions?
  2. In fact, android use always use assistive tools. What is the speciality of the assistive tools in MacOS?
Reviewer_avwkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

AccessWorld evaluates computer use agents operating native macOS assistive technologies. The authors hold 124 tasks, initial environments, and final-state graders fixed across standard mouse-and-keyboard interaction, VoiceOver, 2× Zoom, 5× Zoom, and Voice Control. Results from five models show that the interface changes model rankings, task success, and interaction cost. Some configurations approach or exceed standard computer-use success with larger action budgets, but require more interaction. Voice Control performs worst for every model.

Critical strengths
  1. Native assistive technologies form part of the actual observation–action loop. For example, VoiceOver exposes neither screenshots nor pointer tools, addressing a capability gap that standard GUI evaluation does not capture.
  2. Comparing interfaces on shared tasks and graders, while reporting costs and success-versus-budget curves, shows that standard computer-use scores do not characterize assistive-technology operation. This is a clear research contribution.
Critical weaknesses
  1. The tasks lack validation by assistive-technology users, leaving agent limitations confounded with applications that cannot be operated through the assigned interface. This limits the evidence available for the proposed accessibility-auditing use.
  2. Low Voice Control success is associated with budget exhaustion and task type, but the analysis does not distinguish command generation, speech recognition, and application operability. Explanations of the underlying difficulty and remedies are therefore less well supported than the descriptive results.
Questions for authors
  1. Did manual task validation include complete operation through the assigned assistive technologies, or only establish feasibility in the reference environment?
  2. Could a sample of existing Voice Control failure traces distinguish command-generation errors, recognition errors, and application barriers, and report the leading causes?
Reviewer_3CgrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper is in general a benchmark paper which evaluates compute-use agents through native macOS assistive technologies. It combines 124 tasks adapted from macOSWorld and MacArena, covering 33 applications and system interfaces. The main contribution is the evaluation infrastructure and some empirical finding and there contains no new agent algorithms.

Critical strengths
    1. this paper is timely. AccessWorld places efforts in understanding different interactions of an agent for compute-use tasks. The topic itself is sound and interesting.
    1. the analysis goes beyond reporting performances between different input modalities.
Critical weaknesses
    1. The central claim and its supporting evidence are unclear. The benchmark does not sufficiently distinguish agent limitations on different input interface. I do not understand what is the central claim of the authors. The authors should articulate the hypotheses being tested and explain how the experiments support them, rather than leave reviewers to infer the scientific conclusions from the reported scores.
    1. The comparisons lack sufficient validation of interface-specific adaptation. I cannot believe the author consider the experiments fair as they simply put different input into a same system without any effort on adopting them. Without stronger adaptation baselines or diagnostic experiments, the reported differences are difficult to interpret as a fair comparison of agent capabilities across interaction modalities.
Questions for authors
    1. What conclusions do you intend to draw beyond the observation that performance varies across interfaces?
    1. How did you validate that the prompts and agent configuration were appropriate for each interface?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.