- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents AccessWorld, a benchmark of 124 macOS tasks, uses existed questions but changes the tools to VoiceOver, Zoom and Voice Control. Five models are tested under five interaction settings. The main finding is that some assistive settings reach or even pass the success rate of normal interaction, but they need much more interaction to do so. Another finding is that the ranking of models under normal interaction does not carry over to the assistive settings.
Critical strengths
The evaluation is carefully executed and goes beyond headline accuracy: the paper characterises how each assistive-technology condition fails — budget exhaustion under Voice Control, fixed-magnification behaviour under Zoom, and false completion claims in self-reporting — and turns each into a concrete design implication. This diagnostic value is what makes the contribution consequential despite the inherited task pool.
Critical weaknesses
Tasks are only checked to be solvable in the reference environment, not under each assistive setting. It is not interesting that limiting the tools for an agent harms the scores. But we dont need limit the agent. "Phone use" usually uses assistive tools, and I dont think macOS is special. Moreover, "Phone use" uses assistive tools because it is simpler than vision, but PC agent has more paths to control.
Questions for authors
- Is the limited tools harms the informations? i.e. does it change the solvability of the questions?
- In fact, android use always use assistive tools. What is the speciality of the assistive tools in MacOS?