acceptodds
Under review as a conference paper at ICLR 2027

CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

Abstract

To address the brittleness of mobile GUI agents on unseen applications, we study test-time adaptation using limited target interaction without target demonstrations. We introduce CoAdapt-GUI, a test-time adaptation (TTA) framework that jointly adapts structured workflow context and policy from the agent’s own target-app rollouts and rewards. This context guides target adaptation with transferable pro- cedures, failure modes, and verification rules while excluding app-bound source state. For policy adaptation, task–context-matched group-relative optimization updates a LoRA adapter on a frozen vision–language model. Across two unseen- app evaluations, CoAdapt-GUI reaches 45.0% on AndroidWorld-Generalization, compared with 37.08% for the reproduced Policy-Only TTA baseline, and raises AndroidWorld Plus performance from 37.9% to 52.6%. These results show that transfer-constrained workflow context provides substantial gains and that joint policy adaptation further improves held-out performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile GUI Agent, Test-time adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9upNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoAdapt-GUI adapts both a transferable workflow context and a GUI policy to a new application. It evolves context variants with rollout feedback while updating LoRA policy parameters using task-context-matched groups. The paper evaluates unseen-app adaptation in Android-Generalization and AndroidWorld Plus, with the latter separating adaptation and evaluation task templates. Joint adaptation improves reported success rates over static transfer and either adaptation component alone.

Critical strengths
  1. The method addresses a practical mismatch between reusable workflow knowledge and the behavior of a policy executing that knowledge. The immutable source/mutable target distinction and task-context-matched update groups are sensible design choices, and the implementation is described in sufficient detail to understand their intended role.

  2. The experiments include relevant component baselines and a stronger template-disjoint evaluation in AndroidWorld Plus. Joint adaptation reaches 52.57% success versus 47.24% for context-only and 40.38% for policy-only adaptation; the shared/novel-category breakdown makes the remaining transfer difficulty visible. Freezing adaptation before evaluation is an important strength.

Critical weaknesses
  1. The evidence establishes a useful joint system more clearly than it establishes the proposed mechanisms. Removing all context or policy adaptation does not isolate the transfer constraints, the context-matched credit assignment, or the population/rating mechanism. Appendix E primarily compares the main configurations rather than ablating those choices. Since joint prompt/weight adaptation is already discussed in the related work, identifying which GUI-specific design contributes beyond simply running both components is central to the methodological contribution.

  2. The adaptation and selection comparisons need clearer accounting. Appendix B.3 states that nominal update counts do not constitute matched compute, while joint adaptation includes context generation/reflection and candidate evaluation. The reported variation is also insufficiently explicit about independently repeated adaptation runs versus repeated evaluation of adapted checkpoints. Finally, Appendix C.3 retains historical context ratings across policy changes without final-policy reranking; this leaves the selected context's compatibility with the final policy empirically unresolved. These limitations make the size and reproducibility of the claimed co-adaptation advantage difficult to assess.

Questions for authors
  1. Under the same environment-interaction budget, what happens when task-context grouping is replaced by task-only grouping, and when the workflow transfer constraints are removed? A focused ablation of the claimed mechanisms would materially affect my assessment.

  2. Please report realized environment calls, optimizer updates, reflection/generation calls, and adaptation seeds for each main baseline. How much of the joint gain remains across independent adaptation runs?

  3. Does reevaluating/reranking candidate contexts under the final policy materially change the chosen context or test performance?

Reviewer_WNovOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This work introduces a test-time adaptation framework that updates separate workflow-context and policy states using only the agent’s own target-app rollouts and task rewards, without target demonstrations or access to held-out evaluation signals.
  • This work separates transferable workflow knowledge from app-bound source state and coordinates two reward-guided updates from the same target interactions. Validated reward differences revise the workflow context, while policy credit is computed only among rollouts sharing the same task and context condition.
  • This work evaluates CoAdapt-GUI on the released AndroidWorld-Generalization unseen-app split and constructs AndroidWorld Plus to hold out entire task types during target adaptation. CoAdapt-GUI reaches 45.0% and 52.6% in the two settings, respectively, achieving the best overall result in both settings.
Critical strengths
  • Clear motivation. CoAdapt-GUI is a test-time adaptation framework that updates the workflow context and policy on unseen applications using the agent’s own target rollouts and task rewards.

  • During the (TTA) phase, CoAdapt-GUI requires no human demonstrations; instead, it leverages autonomous interaction with the target application and reward feedback to jointly optimize the external workflow context and the underlying LoRA.

Critical weaknesses
  • CoAdapt-GUI relies on task-level rewards to update its policy. But it appears unachievable in practical applications.

  • The experiments are overly simplistic; it is unclear whether the proposed CoAdapt-GUI possesses sufficient generalizability or practical application value. Furthermore, there is a lack of comparison with similar methods or general-purpose UI agents.

  • The paper lacks the necessary ablation studies to substantiate its contributions. For example, the design of the reward.

  • Minor issue:

    • "Base Policy" is not an easy-to-understand code name.
    • The layout of Table 1 is not aesthetically pleasing and does not conform to academic writing standards.
Questions for authors
  • Please refer to the weakness.
Reviewer_AMbuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoAdapt-GUI, which adapts workflow context and a LoRA policy adapter to unseen GUI applications using the agent’s own rollouts and executable rewards. It restricts transferred workflows to abstract procedures, failure patterns, and completion checks, and computes policy advantages within groups sharing the same task and context.

Critical strengths
  1. The writing is clear.
  2. Joint adaptation consistently improves over context-only and policy-only adaptation in both evaluation settings.
  3. The evaluation covers both new task instances and held-out task templates in unseen applications.
Critical weaknesses
  1. The ablations show that joint adaptation helps, but do not isolate the benefits of transfer constraints or task-context matching over a simpler joint method.
  2. The interaction budget is unclear. CoAdapt-GUI can evaluate two contexts per round, while policy-only uses a fixed context. Actual rollout counts are not reported, making it difficult to verify whether the budgets are matched.
  3. The final context is selected using ratings accumulated under changing policies, without reranking the candidates under the final policy. These ratings may not reflect how well the contexts work with the final adapter.
Questions for authors
  1. With the same reflector and interaction budget, how does the joint method perform when transfer constraints are removed? What happens when policy advantages are computed across context variants of the same task, rather than within matched task-context groups?
  2. Please clarify the actual target rollout counts and effective policy-update counts for each adaptive configuration. In AndroidWorld Plus, how is the interaction budget matched between the fixed-context policy-only baseline and rounds that evaluate two context variants?

Comments on the paper

1 comment
Reviewer_WNovOpenReview ↗

Official Comment by Reviewer WNov

  • The rationale for using UI-TARS-7B on AndroidWorld-Generalization but Qwen3-VL-8B-Instruct on AndroidWorld Plus is unclear. Please justify this choice and evaluate both backbones on the same benchmark to assess whether the improvements are consistent across models.

  • Why not just use "Base Policy" as the model name?

  • Unlike Table 2, some models in Table 1 span two rows. This is the reason for my question; if the authors do not consider this an issue—or if it was intentional—then please feel free to disregard this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.