- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OSGuard evaluates benign-task computer-use safety at two granularities: 324 human-labeled pre-execution action judgments and 45 environment-modified OSWorld tasks with state-based safety checks. The best offline guardrail reaches 79.9% accuracy, but online guarding leaves safe success at 62.2% and reduces unsafe completion from 37.8% to 33.3%. The study highlights the gap between local classification and full-task safety.
Critical strengths
- Unchanged benign instructions and explicit preservation checks expose collateral harm that nominal OSWorld success would miss.
- Connecting action judgments with guarded execution makes the offline–online gap concrete, and the paper transparently reports both corrected and newly terminated tasks.
Critical weaknesses
- Benchmark contribution: The useful setting is benign-task collateral harm, but the paper mainly combines existing action classification with environment modifications. Its added coverage over MisActBench, BLIND-ACT, and other computer-use safety benchmarks is described qualitatively. A common-task or failure-category comparison would better establish which consequential failures OSGuard newly captures.
- Dataset validity: Construction source is strongly associated with label: all 49 standard-execution items are allowed, and neither proposer-generated source contains allowed items (Table 1). This risks rewarding source-related cues. Annotator numbers, agreement, and adjudication are also unreported, making subjective unsafe/unrelated boundaries difficult to audit.
- Execution evidence: Online tests use one executor and one guardrail on 45 tasks. The change from 17 to 15 unsafe completions involves only two fewer tasks, while safe success stays at 28/45 and one previously safe task is terminated. Without repeated runs or other pairings, these results are insufficient to establish a general offline–online relationship.
- Measurement protocol: Candidate units mix primitive actions and short plans without a granularity-specific analysis; this changes what risk can be inferred before execution. Item-level bootstrap intervals also do not address shared-task/trajectory dependence, and state-based checks need to distinguish transient violations from final-state preservation. These choices can materially affect benchmark scores.
Questions for authors
- Which failure categories are newly captured relative to existing benchmarks, and how do source-balanced scores and independent annotation agreement change the conclusions?
- Are the online results stable across repeated runs and other executor/guardrail pairs, with task-clustered paired uncertainty?
- How are primitive versus compound candidates labeled and analyzed, and which safety checks monitor intermediate states rather than only terminal state?