acceptodds
Withdrawn from ICLR 2027

AsyPO: Asymmetric Post-Training for GUI Agents with Segment-Level Reasoning Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Graphical user interface (GUI) agents require both executable actions and reasoning, but existing outcome-based post-training provides only coarse supervision for the latter. Existing post-training methods mainly include reinforcement learning (RL) and on-policy self-distillation (OPSD). Although OPSD enables process-level supervision through teacher–student token probabilities, applying it independently to long reasoning spans can distort their global structure. We propose an asymmetric post-training framework that combines action-level Group Relative Policy Optimization with relative segment-level reasoning guidance. Our method aggregates teacher guidance within reasoning segments, centers the resulting signals across each reasoning span, and selectively applies them through negative-advantage gating. This provides targeted reasoning supervision while preserving the overall structure of model-generated reasoning. Experiments on OSWorld and OmniACT demonstrate consistent improvements over the evaluated baselines at both 4B and 8B model scales.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-Use Agents, On-Policy Self-Distillation, Segment-Level Reasoning Guidance, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SS67OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on graphical user interface (GUI) agents, and the authors introduce AsyPO: an asymmetric post-training methodology with segment-level reasoning guidance. To the best of my understanding, the proposed method applies GRPO to action tokens while using OPSD for segment-level reasoning guidance. The authors test their proposed method on two models (Qwen3-VL-4B-Thinking and Qwen3-VL-8B-Thinking) on two datasets (OSWorld and OmniACT), demonstrating improvements over the provided baselines.

Critical strengths

S1) Strong suite of baselines: I appreciate that the authors took the time to test a variety of baselines. In my opinion, these are pretty helpful for comparison

S2) Informative Appendix: I believe that the authros ran many helpful experiments to understand their method better, especially when looking at the various ablations provided in the appendix

Critical weaknesses

Critical

W1) No uncertainty reported: Unfortunately, the authors report no uncertainty or confidence in the main manuscript. In their defence, they provide it in the Appendix, but only for Table 7, which helps clarify the method. Their method performs better than SDAR, but there are several instances of overlapping standard deviations (I assume? I did not find a description). Naturally, I don't expect the authors to run their experiments multiple times, but I would at least encourage them to report, e.g. a bootstrapped confidence interval over the test set, as that can already give us an indication of how much spread there is in the results, and how statistically meaningful they are.

Soft

W2) Performance of the method seems very reliant on Hyperparameters: Looking more closely at the ablations in the Appendix, I see that performance on OmniACT can range from 33.1 to 38.2 (the reported score). In this case, most baselines outperform 33.1, except base and OPSD. To me, this suggests that performance is quite dependent on the right hyperparameter choice. This is not necessarily a bad thing, but I wonder if the same amount of effort went into finding good hyperparameters for the baselines, and how much variability there is in the results (re-emphasising the point above)

Questions for authors

Q1) Hyperparameters chosen for different baselines: How did you choose the baselines' hyperparameters to ensure a fair comparison with AsyPO?

Q2) Trajectory-Level Baselines for OmniACT: I might have missed something, but why are they not reported for OmniACT?

Reviewer_DWgYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AsyPO trains GUI agents with asymmetric supervision for interleaved reasoning and action outputs. Action tokens receive GRPO updates from a GUI reward, while reasoning tokens are partitioned into contiguous segments (eight-token blocks by default) and weighted by a privileged-teacher log-probability gap. Segment weights are centered within a reasoning span and applied only to trajectories with negative group-relative advantage (§4.1, pp. 4–5). Experiments cover OSWorld and OmniACT at 4B and 8B scales. The central empirical claim is that this combination improves task success over action-only GRPO and prior distillation variants; the broader “structure preservation” interpretation is less directly established.

Critical strengths

AsyPO applies outcome-based GRPO to executable actions and relative teacher guidance to reasoning, addressing the different supervision needs of these outputs.

Critical weaknesses
  1. The reward can systematically rank an incorrect action above a valid alternative. Equation (3) and Appendix B.3 assign a functional score of 1 whenever the predicted action type matches the reference, regardless of argument correctness. Without penalties, an incorrect same-type click can therefore receive approximately 0.60, while a fully valid different-type alternative receives at most 0.50. This undermines the claimed accommodation of alternative GUI paths and raises a consequential concern about what the policy is learning to optimize.

  2. The experiments do not establish the proposed reasoning mechanism. Better benchmark performance does not demonstrate that the segment weights identify useful reasoning or preserve its structure. Negative-advantage gating also changes the amount of auxiliary supervision, while segmentation and normalization change its effective strength. Without matched controls, the gains could reflect more conservative regularization rather than meaningful reasoning credit assignment. This limits confidence in the paper’s central scientific explanation, even if the engineering gains are reproducible.

Questions for authors
  1. Can the reward rank an incorrect action above a valid alternative? Equation (3) and Appendix B.3 assign a functional score of 1 whenever the action type matches the reference, regardless of argument correctness. Without penalties, an incorrect same-type click can therefore receive approximately 0.60, while a fully valid different-type alternative receives at most 0.50. Does the implementation actually produce this ranking? If additional checks prevent it, please specify them. 
  2. Do the ablations in Table 3 isolate individual components? Please specify the token masks, gating, centering, normalization, auxiliary coefficient, and loss reduction used in each row. In particular, Table 1 describes GRPO+OPSD as operating on the full response, whereas Table 3 uses the corresponding configuration to motivate segmentation and centering. Were supervision scope and effective auxiliary strength held constant? 
  3. What is the exact segment-weight normalization procedure? Appendix B.4 states that the normalization statistic, numerical stabilizer, and remainder-packing behavior depend on the unavailable trainer implementation. Please provide the complete procedure, including the normalization scope and statistic, handling of incomplete segments, and ordering of clipping and re-centering. These choices directly determine the optimization objective and the effective meaning of β = 0.05. 
  4. How is the privileged teacher instantiated and updated? Which checkpoint supplies the teacher probabilities? Is the teacher frozen, periodically refreshed, or updated together with the student? Please also specify the complete teacher prompt and how the gold action is incorporated. The reward judge is identified, but the teacher is a separate component whose configuration is essential to interpreting the proposed guidance.  
  5. What is the correct gate-off result in Figure 4? The overall panel annotates a 4.33-point improvement from gate-off to gate-on. Given the full method’s approximately 38.21% score, this implies approximately 33.88% for gate-off, whereas the text reports 34.4%. Please reconcile these values. Also, do the “no segment signal” domain results and Table 3’s action-only GRPO results come from different runs?
Reviewer_ZmaWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AsyPO, an asymmetric post-training framework for GUI agents that assigns different objectives to executable actions and reasoning tokens. Action tokens are optimized with GRPO using a composite GUI reward. Reasoning tokens receive guidance derived from privileged-teacher versus rollout-policy log-probability differences, averaged within fixed-length segments and centered within each reasoning span. This guidance is applied only to samples with negative group-relative advantages. Experiments on OSWorld and OmniACT with 4B and 8B backbones show improvements over the evaluated baselines, including SDAR. The paper also presents component ablations and results from eight independent training runs.

Critical strengths

This paper shows the following strengths:

  1. A well-motivated distinction between action and reasoning supervision. Action rewards do not necessarily identify which parts of a reasoning trace are useful. Separating the direct optimization objectives for these two output components is a sensible response to this credit-assignment problem. The proposed segment weighting is relatively simple and does not require training a separate process reward model.
  2. Consistent empirical gains supported by multiple evaluations. AsyPO improves OSWorld success over SDAR from 33.3% to 37.1% at 4B and from 36.0% to 38.2% at 8B, with improvements on both reported OmniACT metrics. The eight-run results strengthen the evidence beyond a single training seed, while the ablations provide useful evidence that unprocessed or indiscriminately applied teacher guidance can be detrimental.
Critical weaknesses
  1. The incremental contribution over closely related RL–distillation methods is not sufficiently isolated, and the mechanistic interpretation remains under-supported.
  2. The functional interpretation of the reward is insufficiently justified. When the predicted action type matches the reference, the functional-action score is fixed to one regardless of argument correctness. Consequently, an incorrect click can receive substantial credit solely from matching the action type. This does not necessarily invalidate training, but it raises concerns about how faithfully the reward ranks task progress and functionally equivalent alternatives.
  3. The method uses fixed-length token blocks rather than semantic reasoning units. The experiments therefore support local smoothing of teacher guidance more directly than structure-aware reasoning optimization. The paper should clarify this distinction and moderate claims about preserving global reasoning structure.
Questions for authors

Q1. Clarification of novelty and comparability: How exactly does AsyPO differ from the implemented SDAR baseline in teacher construction, gating condition, GRPO token scope, distillation token scope, and weight normalization? Were the reward function, initialization, and training budget identical? A compact objective-level comparison would materially help assess the incremental contribution.

Q2. Evidence affecting attribution: Do you have a comparison against random gating with a matched number of selected samples and comparable effective auxiliary-loss strength, or against an appropriately downweighted ungated objective? Such evidence would help distinguish the benefit of negative-advantage selection from the benefit of reducing distillation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.