- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SAFEST, a rule-guided framework for automatically synthesizing executable safety training data for Computer-Use Agents (CUAs), addressing the gap between text-based safety alignment and safe behavior in real GUI environments. The framework generates both direct and indirect harmful tasks, constructs executable environments, and collects validated safe-execution trajectories for supervised fine-tuning. The resulting dataset contains 2,724 tasks across 8 applications and 43 harmful categories. Experiments on Qwen3-VL-8B and EvoCUA-8B demonstrate substantial improvements in average safety rates across four benchmarks, while maintaining relatively low over-refusal rates and preserving general GUI capabilities.
Critical strengths
-
Valuable training resources for CUA safety. The paper introduces a useful framework and dataset for improving CUA safety alignment, providing executable safety tasks and validated safe-execution trajectories that can support future research on safer CUAs.
-
Comprehensive experimental evaluation. The paper evaluates its approach across multiple CUA safety benchmarks and model backbones, demonstrating substantial safety improvements. The additional analysis of over-refusal and general GUI capabilities provides valuable insights into the trade-off between safety and utility.
Critical weaknesses
-
Limited exploration of safety alignment methods. The paper exclusively relies on supervised fine-tuning (SFT) without investigating alternative alignment methods, such as preference-based alignment (e.g., DPO) or reinforcement learning with verifiable rewards (RLVR). It remains unclear how different training paradigms would influence safety performance and the trade-off between safety and utility.
-
Potential distribution overlap between training and evaluation. The synthesized training tasks and evaluation benchmarks largely share similar OSWorld-style environments, applications, and task distributions, raising concerns about potential distribution overlap. Although the authors conduct instruction-level similarity analysis, this does not fully rule out overlap in task structures and underlying safety scenarios, potentially limiting the generalizability of the reported improvements to more diverse, out-of-distribution CUA environments.
Questions for authors
N/A
3 comments
Dear Reviewer,
We would like to gently follow up on our response above before the rebuttal period ends. Regarding the exploration of alignment methods beyond SFT, we provided an additional on-policy self-distillation experiment demonstrating that SAFEST’s structured execution plans can support a different alignment paradigm. To examine generalization beyond OSWorld-style environments, we also evaluated SAFEST on ADEPTS, an independently constructed benchmark covering approximately 220 applications across desktop and mobile platforms, and observed consistent safety improvements on both platforms.
We hope these additional experiments address your two main concerns. We would be grateful to hear whether you find the responses satisfactory, or whether any questions remain. Your feedback would be very valuable to us as we prepare the revision.
Thank you again for recognizing the value of our work and for your constructive comments.
Best regards,
The Authors
Response to Rebuttal
Thanks for providing the new results, which solve my concerns to a large extent. I choose to maintain my positive assessment of the paper.
We thank the reviewers for their detailed and constructive comments. Below we address each point in turn.
W1: Extending SAFEST beyond SFT
Thank you for raising the question of whether SAFEST can support alignment methods beyond SFT. SAFEST was originally constructed as an SFT resource of task instructions and validated safe-execution trajectories; it does not provide preference pairs or online reward signals for directly applying methods such as DPO or RLVR. We nevertheless explored on-policy self-distillation (OPSD), an OPD-based alignment method, using each task's execution plan as privileged information available to the teacher model. This lets us test whether SAFEST's structured supervision can be used in a different alignment paradigm.
These results show that SAFEST execution plans can support an OPD-based alignment approach, extending their use beyond standard SFT. We will add this experiment in the revision.
W2: Generalization to diverse, out-of-distribution CUA environments
Thank you for noting that overlap in OSWorld-style environments may limit generalization. To test transfer beyond SAFEST's training distribution, we evaluated Qwen3-VL-8B on ADEPTS, an independently constructed benchmark spanning mobile and desktop applications. Unlike SAFEST's executable OSWorld tasks across 8 desktop applications, ADEPTS uses paired benign/malicious tasks with threats embedded in screenshots and covers roughly 220 applications.
SAFEST improves safe rate on both ADEPTS platforms, providing evidence that its safety gains transfer to visually grounded tasks across a broader application and platform distribution. We will include this OOD evaluation in the revised paper.