GUI-SKE: Structured Knowledge Evolution for Non-Parametric GUI Agent Adaptation

GUI agents powered by multimodal large language models show promise for mobile task automation, yet can struggle in unseen applications where navigation structures and element-level interaction behaviors differ from prior experience. Parametric adaptation incurs additional optimization costs, while acquiring environment-specific knowledge and refining it using execution feedback remain challenges for non-parametric approaches. We introduce GUI-SKE, a non-parametric Structured Knowledge Evolution framework that adapts GUI agents by constructing and refining external knowledge while keeping model parameters fixed. GUI-SKE performs two-level exploration to acquire GUI-specific interaction evidence at different granularities and organizes it into linked functional, transition, and operational knowledge components. It derives pseudo-tasks with verification requirements from this evidence and uses rollout feedback as weak supervision for knowledge refinement. Group-wise trajectory comparison combines task-success and component-specific process feedback to guide targeted edits to the corresponding knowledge entries. Experiments on AndroidWorld, SPA-Bench, and our proposed ProMobile-Bench show that GUI-SKE achieves higher task success rates than the baselines included in our comparisons.