acceptodds
Desk rejected at ICLR 2027

AdapGuide: Turning Tutorial Knowledge into Evolving Guidance for GUI Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI agents rely on external tutorials to acquire procedural knowledge for unfamiliar software tasks. Existing methods retrieve relevant tutorials and convert them into action guidance. Yet tutorial guidance remains tied to its source interface and example, so it may not fit the current task and can become invalid when actions produce unexpected effects. Fixed guidance then propagates local errors through subsequent stages of task execution and may mistake an incorrect state for progress. To bridge this gap, we introduce AdapGuide, a training-free framework that reconstructs reusable operation–effect relations and revises task-specific guidance throughout execution. Each relation links a configured operation to its applicable conditions and expected interface change, enabling AdapGuide to bind tutorial knowledge to the objects and settings required by the current instruction and interface. During execution, mismatches between observed and expected effects trigger revisions to object bindings, operation settings, and subsequent stages. Experimental results across three benchmarks demonstrate the effectiveness and generalizability of AdapGuide, with average task score gains of 8.08–15.86 pp across five backbones on OSWorld-Verified, 14.73 pp on Windows Agent Arena, and 12.47 pp on DeskCraft.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agents, Procedural Knowledge Transfer, Operation-Effect Relations, Online Guidance Revision

Related papers

Discussion (0)

Sign in to comment.