acceptodds
Withdrawn from ICLR 2027

ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online training enables computer-use agents (CUAs) to improve through interaction with executable environments. However, existing methods primarily rely on sparse outcome rewards, which provide no supervision for intermediate actions. On-policy self-distillation (OPSD) offers token-level learning signals through privileged rescoring, but directly applying it to CUA online training presents two challenges: fixed guidance may become misaligned with the student's current state, and guidance-induced probability shifts may conflict with step-level correctness. We introduce ComputerSD, an online self-distillation method for CUAs that converts real-time feedback from executed GUI transitions into guidance for policy learning. A fine-tuned GUI analyzer produces guidance and a step-level value score after each action; the guidance provides privileged context, while the score regulates the resulting OPSD signals. ComputerSD jointly optimizes token-level OPSD and trajectory-level GRPO in a fully asynchronous training framework. On OSWorld-Verified, ComputerSD outperforms outcome-only GRPO by 1.9 and 4.1 percentage points on the general-purpose Qwen3-VL-8B-Thinking and specialized EvoCUA-8B backbones, respectively. Evaluation in out-of-distribution settings further supports the generalizability of ComputerSD. These results demonstrate the effectiveness of learning from real-time feedback through online self-distillation for CUAs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-using Agent, On-policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_s5SqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an online self-distillation method for computer-use agents. A GUI analyzer examines pre- and post-action observations to provide guidance and a step-level value judgment. The policy then scores the same sampled response with and without this guidance, using the token-level log-probability differences and value gating to construct self-distillation signals. These signals are combined with trajectory-level GRPO to optimize the policy.

Critical strengths
  1. The method uses step-level judgments to regulate self-distillation signals. A value gate checks whether guidance-induced probability shifts align with these judgments, without requiring an additional trainable gating network.
  2. The paper presents a reasonably comprehensive set of ablation studies, with results supporting the contribution of each core component to overall performance.
Critical weaknesses
  1. The method relies on sufficiently reliable guidance and step-level judgments from the GUI analyzer, which is fine-tuned on annotations produced by a stronger proprietary expert. However, agreement with the expert does not establish the correctness of those annotations. An independent assessment of annotation quality, together with an analysis of robustness to erroneous feedback, would strengthen the paper.
  2. Although the paper includes additional prompt-only and GRPO-with-PRM baselines, the comparisons could be broadened to include GiGPO and alternative policy optimization methods such as GSPO, Dr. GRPO, and DAPO. Comparisons under matched backbones, interaction budgets, and training resources would help establish whether the gains extend beyond the chosen GRPO baseline.
  3. Analyzer construction cost. Constructing the analyzer requires collecting 1,776 trajectories, obtaining expert annotations for 34,188 steps, and performing supervised fine-tuning. A more complete accounting of these upfront costs, together with a data-scaling study, would help assess the method’s practical efficiency and the amount of annotated data actually required.
  4. Comparison with direct expert feedback. The reported agreement with expert step-level judgments does not establish comparable guidance quality or downstream policy performance. A controlled comparison between using the proprietary expert directly and using the fine-tuned analyzer would clarify the performance–cost trade-off and provide stronger evidence for the effectiveness of distillation.
Questions for authors
  1. How reliable are the expert annotations, and how robust is the method to incorrect feedback?
  2. How does the method compare with GiGPO, GSPO, Dr. GRPO, and DAPO under matched settings?
  3. What is the total cost of constructing the analyzer, and can comparable performance be achieved with fewer annotated samples?
  4. How does the fine-tuned analyzer compare with direct expert feedback in downstream performance and cost?
Reviewer_WXA4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ComputerSD, an online self-distillation framework for computer-use agents that aims to address the limited credit assignment provided by trajectory-level outcome rewards. The authors identify two limitations of existing on-policy self-distillation methods: fixed guidance may become misaligned with the agent’s actual execution state, and guidance-induced token probability shifts may conflict with step-level correctness.

To address these issues, ComputerSD introduces a fine-tuned GUI analyzer that generates real-time guidance and step-level value judgments based on executed GUI transitions. The guidance is used as privileged context for self-distillation, while the value judgments regulate token-level learning signals through a value gate. The resulting objective is combined with trajectory-level GRPO in an asynchronous training framework.

Experiments on OSWorld-Verified demonstrate improvements over outcome-only GRPO on both Qwen3-VL-8B-Thinking and EvoCUA-8B. Additional evaluations on held-out application categories and WindowsAgentArena suggest improved generalization.

Critical strengths
  1. Credit assignment is particularly challenging in long-horizon GUI tasks, where a failed trajectory may contain many correct intermediate actions. Using real-time environment feedback to provide finer-grained supervision is a meaningful research direction.

  2. The motivation for combining real-time guidance with value-based regulation is reasonable.

  3. ComputerSD improves performance over GRPO on two different 8B backbones, with larger gains on held-out application categories. The improvements on WindowsAgentArena also provide some evidence of cross-platform generalization.

Critical weaknesses
  1. It is unclear whether the improvements mainly come from the stronger expert model rather than the proposed algorithm. The GUI analyzer is fine-tuned on annotations generated by Kimi K3. However, Table 3 shows that without analyzer SFT, ComputerSD achieves 38.0%, almost identical to GRPO’s 37.9%. This raises the concern that much of the improvement may come from distilling the stronger model’s knowledge, rather than from the proposed value-gated self-distillation mechanism itself.

  2. The paper lacks a direct comparison with conventional expert distillation. Since ComputerSD already relies on Kimi K3 to generate expert annotations, a natural alternative would be to directly distill Kimi K3’s knowledge into the policy through SFT or other distillation methods.

  3. Compared with GRPO, ComputerSD introduces an additional GUI analyzer and privileged rescoring at every step. Given this extra computational cost, it would be valuable to compare against methods that instead train an additional critic, like PPO.

Questions for authors
  1. What if we directly use the outcome reward instead of the step-level value score for gating? Could the authors compare this simpler alternative with the proposed value gate to better demonstrate its effectiveness?
Reviewer_TNXNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ComputerSD, an online self-distillation method for computer-use agents. A fine-tuned GUI analyzer generates step-level guidance and value scores from screenshots before and after an executed action. The guidance serves as privileged context for the policy model, while the value scores construct a Value Gate over token-level teacher–student log-probability gaps. The final training objective combines this self-distillation term with outcome-reward-based GRPO.

On OSWorld-Verified, ComputerSD improves task success over GRPO by 1.9 and 4.1 percentage points on Qwen3-VL-8B-Thinking and EvoCUA-8B, respectively. The paper also reports improvements on held-out task categories and WindowsAgentArena. The same-backbone comparisons and component ablations are positive aspects of the empirical evaluation. However, the current experiments demonstrate the utility of the overall system more convincingly than they establish whether the proposed loss works through the claimed mechanism and whether its design is necessary relative to simpler alternatives.

The problem formulation and overall pipeline are generally clear. However, the relationships among token-probability shifts, actual policy updates, and judgments of action correctness are not sufficiently distinguished. In particular, Equations (5)–(8) should specify more explicitly which quantities are held fixed and which receive gradients. The mechanism analysis and controlled experiments isolating the Value Gate's contribution also require further development.

Combining post-action GUI feedback with token-level self-distillation and implementing it in an asynchronous online training pipeline has practical value. Nevertheless, the independent effect and design rationale of the Value Gate, the central algorithmic contribution, are not sufficiently established by the current evidence.

Critical strengths
  1. The motivation that terminal success or failure provides insufficient supervision for intermediate actions in long-horizon GUI tasks is sound. Generating feedback grounded in actual execution outcomes is also a natural approach.

  2. Both a general-purpose model and a computer-use-specialized model improve over GRPO. The paper also reports positive results on held-out task categories and a different operating system.

  3. The component ablations in Table 3 and the gate comparisons in Table 4 are useful. The analyzer and privileged context are used only during training, so deployment does not require an additional module. Asynchronous execution also improves training throughput.

Critical weaknesses

W1. The analysis does not directly validate the central mechanism claimed for the proposed loss.

The central claim is that the Value Gate preserves token-level signals aligned with action value and attenuates conflicting signals, thereby producing more appropriate policy updates. However, the current analysis presents indirect training diagnostics rather than directly testing this claim.

Figures 1(b,c) diagnose problems with existing signals but do not show how much the proposed method actually resolves those problems during learning. The teacher–student gap and entropy measurements in Figures 6 and 8 demonstrate differences in training dynamics, but whether moving closer to the teacher reinforces correct actions is a separate question. Figure 7 likewise characterizes stability under different self-distillation coefficients without explaining the distinctive benefit of value-based gating.

A more direct test would independently assess correct and incorrect actions and measure how each method's actual update changes their probabilities, using the same states and samples. Using only the judgments of the analyzer that generates the gate as the evaluation criterion would risk circular validation.

W2. The comparisons do not sufficiently isolate the contribution of the Value Gate.

The paper includes comparisons against GRPO, several gate variants, and a PRM baseline in the appendix; therefore, it would be inaccurate to say that comparative experiments are absent. Nevertheless, these comparisons do not sufficiently establish the necessity of the proposed token-level objective.

The GRPO with PRM baseline in the appendix sums step-level scores into a trajectory-level reward. It therefore compresses fine-grained feedback back into trajectory-level supervision and does not substitute for a comparison against a strong step-level learning method that directly uses the same feedback.

The SDAR, hard, and reverse gate comparisons in Table 4 are useful, but they do not separate the effect of meaningful action-value information from the effect of reducing the overall auxiliary update magnitude. Comparisons using the same guidance and analyzer outputs, with constant or random gates matched for mean weight or update magnitude, would help determine whether the improvement comes from value-dependent selection or merely from adjusting training strength.

The concern is not the omission of a comparison against a particular recent method, but the lack of sufficient controlled baselines to establish why the proposed design is necessary.

W3. The gradient path of the objective and the role of the value score are unclear.

In Equation (6), the action score is used in the gate input, stδt,ls_t\delta_{t,l}, and the same product also appears in the objective in Equation (7). The action score can therefore affect not only gate selection but also the update direction. These two roles should be distinguished, explained, and evaluated.

In particular, Appendix B states that log-probability gaps and gate weights are computed and cached, but it does not sufficiently clarify which quantities are recomputed during training or where stop-gradient is applied. This is not merely an implementation detail: it determines the meaning of the objective actually being optimized. An explicit gradient expression or short pseudocode is needed. The current description does not establish that the method is incorrect, but it makes the proposed loss difficult to interpret precisely and reproduce.

W4. Uncertainty in the main performance gains is insufficiently reported.

Table 1 reports averages over three evaluation runs without standard deviations or confidence intervals. In particular, the improvement over GRPO on the Qwen backbone is 1.9 percentage points, and the difference from the SDAR gate in Table 4 is 1.7 percentage points. Assessing the consistency of these differences requires reporting evaluation variability and, where possible, distinguishing it from variability across independent training runs. This is not a claim that the results are statistically insignificant; rather, the available information is insufficient to assess the reliability of the gains.

Questions for authors
  1. Mechanism validation: Can the authors show, using an independent assessment of action correctness, whether the Value Gate produces more updates that increase the probability of correct actions and decrease the probability of incorrect actions? A small-scale update analysis using the same checkpoint and batch would be informative.

  2. Controlled baselines: Do the improvements persist against a step-level baseline that directly uses action scores from the same analyzer and guidance, or against constant/random gates matched for mean weight or update magnitude?

  3. Objective clarification: In Equations (5)–(8), which of the teacher log-probabilities, student log-probabilities, log-probability gaps, and gate weights receive gradients? How are cached signals used in the actual training loss, and does the “w/o Value Gate” variant retain weighting by sts_t?

  4. Result consistency: Can the authors provide per-run results or uncertainty estimates for the main comparisons and gate comparisons? Please clarify whether the three reported runs are repeated evaluations of the same trained checkpoint or also involve independent training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.