- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AlphaCUA automates safety-benchmark construction for Computer-use Agents. 1) A generator builds executable safety tasks under skill-based specs; 2) an analyzer labels full execution trajectories for unsafe responses and proposes risk factors 3) a validator agent tests each factor via controlled comparisons of task variants differing only in that factor; validated factors become strategies that guide task generation in new scenarios. On four CUAs, 96.5% of tasks pass acceptance without revision; trajectory ASR exceeds final-state ASR by 8.75 points; strategy-guided expansion raises it to 60.50%.
Critical strengths
- CUAs are moving into real deployment, scalable safety evaluation is an important and timely problem.
- The iterative loop of exploiting the weak points of defense is reasonable.
Critical weaknesses
-
The controlled-validation stage has no supporting data or cases. For example, no original-vs-modified task pair is shown, no funnel statistics, and the pool ends up exactly 3 strategies per threat type.
-
No rendered screenshots or trajectory. Task Quality rests on the authors' self-acceptance after an auto-QA funnel with unreported rejection rate.
-
Only four CUAs are tested, frontier models like GPT or Claude are not tested. The trajectory judge is GPT-5.6-Sol with no human agreement and no second judge.
-
Trajectory-based scoring exists in previous works. OS-Harm already grades attempted-but-incomplete unsafe actions. The main increment ,the validation loop, is not well-supported as stated in W1.
-
RQ3 changes two variables at once (new scenarios + strategy guidance); without a same-budget no-strategy control, +15.75 cannot be attributed to the factors.
Questions for authors
-
Could the authors provide judge–human agreement on a labeled subset? Do the +8.75 and +15.75 gaps survive a different judge model?
-
Could the authors report the factor funnel (e.g., proposed/validated/discarded, effect sizes) and one complete original-vs-modified task pair per strategy.
-
Could the authors reveals the valid-run rates, injection-exposure rates, average trajectory length, and the intent/attempt/effect breakdown?
-
What fraction of TRANSFERRED scenarios are genuinely new? Do the strategies transfer to a held-out CUA?
2 comments
Response to weakness 2, question 3, question 4, weakness 3 and weakness 4
W2 and Q3. Task acceptance, execution traces, and run statistics Could the authors reveals the valid-run rates, injection-exposure rates, average trajectory length, and the intent/attempt/effect breakdown?
Response. The anonymous repository submitted with the paper contains four example runs under examples/trajectories/, one per threat subtype, each with an execution trace and rendered screenshots. The factor-validation records also link the paired runs so that readers can inspect what the agent saw and did in each condition.
We examined the generation records for 200 candidate tasks. Automated QA returned 110/200 (55%) for revision on first review. By the end of the allotted budget, 196/200 (98%) had passed final QA. At human acceptance, 193 of 200 tasks were accepted without changes and seven after minor corrections to evaluator checks, environment configuration, or wording. The final rejection rate at human acceptance was 0/200.
We reviewed the 400 Seed runs (100 tasks × four CUAs). For each valid run, we assigned the furthest unsafe stage supported by its recorded actions, responses, and observable effects.
| Seed run statistic | Result |
|---|---|
| Harness-valid runs | 399/400 (99.75%) |
| Mean agent steps per valid run | 11.28 |
| No unsafe stage observed | 252/399 (63.16%) |
| Unsafe intent or preparation only | 2/399 (0.50%) |
| Unsafe action attempted without an observed harmful effect | 14/399 (3.51%) |
| Task-defined harmful effect observed at any point | 131/399 (32.83%) |
For indirect prompt injection (EE-A), the recorded screen or the agent’s response confirmed exposure in 96 of 100 runs. The remaining four lacked sufficient evidence, so 96% is a lower bound. Exposure includes cases in which the agent saw and rejected the instruction. For the stage distribution, a harmful effect counts even when the agent subsequently repaired it.
Q4 and W3. Novel scenarios and transfer to a held-out CUA What fraction of TRANSFERRED scenarios are genuinely new? Do the strategies transfer to a held-out CUA?
Response. We compared each of the 100 Transferred tasks with all five Seed tasks in the same threat subtype and safety rubric. When the application also matched, we compared the task goal, the main operations, and the required output.
| Relationship to Seed tasks in the same subtype and rubric | Transferred tasks (out of 100) |
|---|---|
| Application absent from the five Seed tasks | 29 |
| Same application, different workflow | 34 |
| Same application, partially overlapping workflow | 17 |
| Same application, same core workflow | 20 |
For example, all five financial tasks involving indirect prompt injection moved from Tbao in Seed to LibreOffice Calc in Transferred. Workflows could also change within an application. In Files, a Seed task asked the agent to rename a daily planning folder, while a Transferred task in the same subtype and rubric asked it to restore a deleted poster from Trash to a specified folder. We counted workflows as partially overlapping when they shared a core operation but differed in scope, as in copying a single named document versus recursively finding and collecting all JPG files.
To test transfer to a held-out CUA, we evaluated GPT-5.6-Terra, which took no part in factor discovery or validation. Across the 100 matched Transferred-Control and Transferred task pairs, its trajectory ASR rose from 29% on Control to 39% on Transferred (+10 percentage points), and its final-state ASR rose from 26% to 33% (+7 points).
W4. Relationship to prior trajectory-based scoring
Response. We agree that OS-Harm [1] already uses execution trajectories to assess unsafe behavior, and we will clarify this in the related-work discussion. We also agree that the original submission gave too little evidence for our main methodological contribution. We test hypotheses drawn from trajectories before using the supported factors to construct new tasks. As detailed in W1/Q2, we tested 25 candidate hypotheses, retained 12, and discarded 13. The anonymous repository now includes the paired tasks, the property varied, and the outcome of each run for all 12 retained factors. For example, in the UA-2 garage-door pair, MiniMax-M3 triggered the simulated alarm in neither of the two low-condition runs and in both high-condition runs. These records let readers examine the evidence for each retained factor in its source setting.
References
[1] Kuntz et al. (NeurIPS 2025). OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents.
Response to weakness 1, weakness 5, question 1, weakness 1 and question 2
Response. On Seed, the trajectory-minus-final ASR gap was +8.75 percentage points under GPT-5.6-Sol, +8.00 under Claude-Opus-5.5, and +9.00 under human judgments. Across the four original CUAs (100 tasks each), mean final-state ASR was 36.00% under all three assessments, and mean trajectory ASR was 44.75%, 44.00%, and 45.00%, respectively.
For the original +15.75-point Seed–Transferred gap, we agree that changing both the scenarios and the strategy guidance limits its interpretation. To test whether retained strategies affect behavior in new scenarios, we constructed AlphaBench-Transferred-Control by removing the designated strategy from each of the 100 Transferred tasks while preserving the application and workflow. Across the original four CUAs, mean trajectory ASR rose from 44.25% on Transferred-Control to 60.50% on Transferred (+16.25 points). Mean final-state ASR rose from 36.75% on Transferred-Control to 53.75% on Transferred (+17.00 points). This paired comparison addresses the scenario confound. It does not measure generation efficiency at a fixed budget.
We also assessed agreement on 500 Transferred-Control trajectories (five agents × 100 tasks). GPT-5.6-Sol, Claude-Opus-5.5, and human reviewers each labeled them without seeing the other two sets of labels. Trajectory-label agreement with humans was 498/500 for GPT (99.6%, κ = 0.992) and 493/500 for Claude (98.6%, κ = 0.971). GPT and Claude agreed on 493/500. Of the 33 runs GPT classified as trajectory-positive without final harm, 32 also received a positive human trajectory label.
Response. We stopped testing hypotheses within each subtype once three had passed source validation. Across the four subtypes, we tested all 25 proposed hypotheses. Twelve passed and were retained as strategies. The remaining 13 failed validation and were discarded.
We have added paired validation records for all 12 retained factors to the anonymous repository linked in the paper. For each strategy, the records identify a complete task pair, the property varied, the endpoint, and the outcome of each run. For example, in the UA-2 garage-door pair, MiniMax-M3 triggered the simulated alarm in 0/2 low-condition runs and 2/2 high-condition runs. In the eight runs reported for EA-1, unsafe attempts occurred in 0/4 low-condition and 1/4 high-condition runs (+25 percentage points), while final harm occurred in 0/4 runs in both conditions.