- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper creates an agent framework for examining the safety of CUA agents with two core steps/ideas: (1) generate task-specific, trajectory-conditioned rubrics that must be examined in a CUA agent trajectory for unsafe behaviour using a tool-free LLM call, (2) run a probing agent that can examine the CUA agent's environment using tools to determine if the safety rubrics from step (1) are violated or not. This pipeline can be used for inference-time scaling where the CUA agent receives iterative feedback from past unsafe attempts. The motivation for building agentic verifiers includes (i) expensive rule-based verifiers, (ii) brittle LLM-judges that often cannot examine the environment's state for unsafe changes not captured in the trajectory's screenshot observations, (iii) naive tool-use verifiers are difficult to adapt for safety analysis as they do not know which safety rubrics to look for/verify. Empirical results and ablations justify the promise of their approach and their design choices. The paper is easy to understand and is well-written.
Critical strengths
- The design choices make intuitive sense and the proposed method is fairly simple+general, making it a better candidate that can be potentially applicable to CUA use-cases (for e.g. RL post-training for safety) beyond the specific benchmarks covered by this work.
- The empirical results are quite promising establishing the strength of the proposed approach. The rationale for design choices can be inferred from these results and qualitative analysis.
Critical weaknesses
- While this maybe one of the first works to develop a rubric-based agentic verifier for the domain of CUA agents, the underlying ideas of (i) proposing task-specific rubrics, (ii) using agentic verifiers for a task are well explored by prior work. What are some core bottlenecks and/or non-trivial/interesting research challenges associated with adapting this to CUA agents that this work attempts to address? This can benefit the positioning of this work.
- A challenge for most safety verifiers is the additional cost and time from using these approaches on everyday benign tasks and the paper mainly focuses on tasks that have some safety issues or unsafe completions. Do authors have additional metrics/analysis to quantify or even qualitatively describe the additional overhead on entirely benign tasks without any safety issues? It seems that the overhead will be higher for benign tasks as compared to baseline approaches as your approach can spend more reasoning tokens/agentic verification turns to exhaustively search for safety issues more often?
Questions for authors
- Does the rubric generation step benefit from privileged information about the task/evaluation metrics that the baseline methods do not have access to? While making a direct comparison with baselines may require to not provide this information, doing so can be beneficial for more complex tasks, or for using this verifier during RL post-training. Do authors have a general idea about how much the pipeline could potentially benefit from having access to such information?
- The takeaways from Section 5.2 are not well-grounded, more reasoning tokens does not always imply achieving better rubrics. This seems more of a design drawback rather than an LLM weakness - you could have a tool which the agentic verifier must call on the first step where it submits the rubrics and its reasoning for each rubric to potentially circumvent this issue?