acceptodds
Under review as a conference paper at ICLR 2027

Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents

Abstract

Graphical User Interface (GUI) agents enable the fulfillment of complex user instructions through multi-turn interactions with software environments, requiring step-wise reasoning and long-horizon memory to guide actions and preserve task-relevant information, respectively. Recent on-policy self-distillation (OPSD) methods have achieved strong performance on GUI grounding, a foundational subtask for GUI agents, owing to dense token-level supervision from privilege-conditioned self-teachers. However, extending existing OPSD methods to multi-turn GUI agents is hindered by self-teachers' limited privilege-following ability and insufficient privileged guidance. In this paper, we introduce GUI-SD-v2, the next version of GUI-SD, which extends OPSD from GUI grounding to multi-turn GUI interaction and addresses key limitations through a two-stage training framework. Specifically, GUI-SD-v2 first strengthens privilege following by jointly optimizing rollouts with and without privileged guidance from the same GUI states. Furthermore, it selectively distills step-specific reasoning and memory guidance through a privilege-conditioned self-teacher, supporting action decisions and the retention of task-relevant information for subsequent interactions. Extensive experiments on two representative GUI-agent benchmarks, AndroidWorld and MobileWorld, show that GUI-SD-v2 compares favorably with existing OPSD baselines while consistently outperforming the evaluated state-of-the-art methods in both Pass@1 and Pass@3 success rates. Code and training data will be publicly released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI Agent, On-policy Self-distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aQrCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to extend on-policy self-distillation from GUI grounding to long-horizon GUI agents. The proposed GUI-SD-v2 uses a two-stage training scheme: first improving the policy’s ability to follow privileged guidance, and then distilling step-level reasoning and memory guidance from a privilege-conditioned self-teacher.

Critical strengths

The paper identifies privilege following and long-horizon reasoning/memory as separate bottlenecks, and the experiments show that both reasoning and memory guidance help, while step-level guidance is substantially better than trajectory-level guidance. This makes the proposed decomposition fairly convincing.

Critical weaknesses

None

Questions for authors

How dependent is GUI-SD-v2 on the quality and scale of the external teacher used to construct privileges? In particular, could the authors report the additional supervision/compute cost of using Kimi-K3?

Reviewer_zcjAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GUI-SD-v2, extending on-policy self-distillation from GUI grounding to multi-turn GUI interaction. The method consists of two stages. First, it jointly trains the policy with and without privileged guidance to improve its ability to use that guidance. Second, it selectively distills action reasoning and interaction memory on trajectories generated by the policy itself. An external model produces the guidance, while the same policy serves as teacher and student under different contexts. The paper evaluates the framework on AndroidWorld and MobileWorld and uses ablations to examine the contributions of the initial training stage, reasoning guidance, and memory guidance.

Critical strengths
  1. A self-teacher may not effectively use privileged information, and supervision of the current action alone does not ensure that information needed by subsequent steps is retained. The initial privilege-following training and separate supervision of reasoning and memory are directly connected to these two problems.
  2. Compared with the authors’ implementation of GUI-SD-v1, AndroidWorld Pass@1 increases from 57.8% to 67.2%, and MobileWorld increases from 18.8% to 25.6%. On AndroidWorld, reasoning-only guidance, memory-only guidance, and their combination achieve 61.2%, 56.0%, and 67.2%, respectively, supporting the complementary roles of the two supervision signals.
Critical weaknesses
  1. PAPO normalizes advantages separately for privileged and original-context samples and jointly optimizes standard GRPO objectives. The second stage applies reverse-KL distillation separately to decision and memory fields. GUI-SD-v1 already uses privileged-context self-distillation, while GHD already uses the next screenshot to supervise a parameter-sharing self-teacher. The paper therefore needs to establish more clearly what independent methodological contribution the additional designs provide over the closest prior work. The ablations demonstrate that the components are useful within this pipeline, but their necessity and advantages over simpler alternatives remain insufficiently established.
  2. The authors control the training-task pools and use Kimi-K3 as the common supervision source, which is valuable. However, identical task pools do not imply matched teacher-call budgets, training-token counts, or environment-interaction budgets. A key comparison would use the same PAPO initialization, student trajectories, and step-specific guidance, and then compare direct SFT with the proposed self-distillation. Furthermore, the GRPO baseline primarily rewards agreement with teacher-provided reference actions. It therefore does not establish whether the proposed method has an advantage over multi-turn RL driven by final task-success rewards. These comparison gaps limit the assessment of the method’s contribution.
Questions for authors

1.Can you compare direct SFT with the proposed self-distillation under the same PAPO initialization, student trajectories, teacher guidance, and training budget? This would help determine whether the gains depend on the proposed distillation mechanism.

2.What motivates using teacher-reference action matching as the reward for the GRPO baseline? For tasks that admit multiple valid action sequences, could this reward restrict baseline performance? Do you have comparisons using environment-level task-success rewards?

3.The memory teacher observes the next screenshot, whereas the student does not observe it when generating memory. How do you prevent the teacher guidance from containing new facts that are only revealed in the next state and cannot be inferred from the current context? Please clarify the constraints used when generating guidance and the ordering of action execution, screenshot acquisition, and memory generation.

4.The appendix specifies three complete evaluation runs. Can you provide the individual results and uncertainty estimates, and clarify whether independent training runs were also conducted? Additionally, does the comparison with GHD in Table 1 use the same evaluation settings?

I consider this a practically useful incremental contribution and currently lean borderline. The targeted supervision for multi-turn interaction and the reported performance improvements are valuable. However, the current experiments do not sufficiently establish the independent value of the additional training designs over simpler alternatives. Strong comparisons under matched budgets and clarification of the temporal ordering of memory supervision would affect my final assessment.

Reviewer_9VveOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GUI-SD-v2 extends on-policy self-distillation to multi-turn GUI navigation through privilege-following optimization followed by selective reasoning and memory distillation, with evaluation on AndroidWorld and MobileWorld.

Critical strengths
  • The evaluation covers two established navigation benchmarks and compares against SFT, GRPO, and OPSD baselines using shared training-task pools and supervision sources.
  • The extensive ablations examine Stage 1 optimization, reasoning and memory guidance, supervision granularity, and alternative guidance providers, providing useful evidence for the design choices.
Critical weaknesses
  • The novelty is incremental because of prior self-distillation works, leaving privilege-following optimization and separate reasoning–memory supervision as the main methodological additions.
  • The paper does not establish whether Stage 2 outperforms continued Stage 1 PAPO on the full training set used in stage 2 with identical Kimi-K3 guidance and a matched training budget.
Questions for authors
  • How does GUI-SD-v2 compare with continuing PAPO on the full training set using the same reasoning and memory guidance under a matched training budget?
  • What are Kimi-K3’s standalone Pass@1 and Pass@3 results on both benchmarks, and what is the gap between the rollout teacher and the final student?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.