- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SafePred, a world-model-based predictive guardrail framework designed to defend against delayed safety risks in computer-using agents. SafePred consists of three components: policy integration, short-term and long-term risk prediction, and hierarchical decision optimization. The framework predicts potential consequences of candidate actions, evaluates risks against structured safety policies, and provides corrective guidance through action regeneration and task-level replanning. The authors also introduce SafePred-Bench, containing 120 tasks designed to evaluate delayed safety risks, and train a lightweight SafePred-8B model through predictive reasoning distillation. Experiments on OS-Harm, WASP, and SafePred-Bench demonstrate improvements in policy compliance and task completion compared with existing reactive guardrails.
Critical strengths
- The paper focuses on an important limitation of reactive guardrails by considering potential downstream consequences of agent actions. The integration of policy-grounded risk prediction with step-level risk guidance and task-level replanning provides a coherent approach to improving both safety and task completion.
- The authors evaluate SafePred on multiple safety benchmarks and report improvements in both policy compliance and task success. The additional SafePred-Bench evaluation, module ablations, inference-budget comparison, and lightweight model distillation further validate the experimental contribution.
Critical weaknesses
- Heavy reliance on world-model capabilities and insufficient validation of long-term prediction reliability. The effectiveness of SafePred fundamentally depends on the underlying ability of the world model to anticipate future consequences. Although Table 7 in Appendix F compares different world-model backbones, the results reveal noticeable performance differences, while SafePred-8B achieves only modest improvements over its Qwen3-8B base model. More importantly, the evaluation primarily focuses on downstream safety and task success, without directly assessing long-term forecast accuracy across different model capacities. As discussed in Section 3.3, long-term prediction is implemented through high-level semantic reasoning. It therefore remains unclear whether smaller world models can reliably anticipate delayed risks and to what extent the improvements arise from genuine predictive capability.
- Presentation quality requires further improvement. Figure 4 contains excessively small font size, making the ablation results difficult to interpret. Additionally, the numbered enumerations in Section 3.1 (Threat Model) and Section 4.1 (Metrics) appear to exhibit unintended hyperlink. Improving these presentation details would enhance the readability and professionalism of the manuscript.
Questions for authors
- Regarding world-model capability, could the authors provide a more detailed analysis of how different model capacities affect short- and long-term risk prediction accuracy? In particular, how reliably can SafePred-8B identify delayed risks compared with larger world models, beyond the downstream performance comparisons in Appendix F?
- Regarding long-term risk evaluation, could the authors provide additional evidence comparing predicted long-term consequences with actual subsequent outcomes? Such an analysis would help establish whether SafePred genuinely anticipates delayed risks before they materialize.
6 comments
W1: Heavy reliance on world-model capabilities and insufficient validation of long-term prediction reliability. The effectiveness of SafePred fundamentally depends on the underlying ability of the world model to anticipate future consequences. Although Table 7 in Appendix F compares different world-model backbones, the results reveal noticeable performance differences, while SafePred-8B achieves only modest improvements over its Qwen3-8B base model. More importantly, the evaluation primarily focuses on downstream safety and task success, without directly assessing long-term forecast accuracy across different model capacities. As discussed in Section 3.3, long-term prediction is implemented through high-level semantic reasoning. It therefore remains unclear whether smaller world models can reliably anticipate delayed risks and to what extent the improvements arise from genuine predictive capability.
Q1: Regarding world-model capability, could the authors provide a more detailed analysis of how different model capacities affect short- and long-term risk prediction accuracy? In particular, how reliably can SafePred-8B identify delayed risks compared with larger world models, beyond the downstream performance comparisons in Appendix F?
Q2: Regarding long-term risk evaluation, could the authors provide additional evidence comparing predicted long-term consequences with actual subsequent outcomes? Such an analysis would help establish whether SafePred genuinely anticipates delayed risks before they materialize.
R1: To directly evaluate World Model prediction capabilities beyond downstream task performance, we extend the evaluation in two aspects: (1) Model-level Prediction Evaluation: We compare Qwen3-8B, SafePred-8B, and Gemini-2.5-Pro on short-term and long-term risk recognition; (2) Execution-grounded Validation: We compare predicted long-term consequences with actual outcomes from controlled execution trajectories.
(1) Model-level Prediction Evaluation.
We conduct two prediction experiments under matched task inputs and evaluation protocols, using each model's corresponding prediction prompt.
(1-a) Short-term Risk Recognition.
We construct 40 samples, including 20 immediate-risk cases and 20 matched safe controls base on SafePred-Bench. All three models receive the same 40 samples, task contexts, policies, and prediction settings. Each model is evaluated once, resulting in 120 valid predictions.
| World Model | Precision ↑ | Recall ↑ | Short-term F1 ↑ | Accuracy ↑ | Safe False Positive Rate ↓ |
|---|---|---|---|---|---|
| Qwen3-8B | 58.6% | 85.0% | 69.4% | 62.5% | 60.0% |
| SafePred-8B | 58.6% | 85.0% | 69.4% | 62.5% | 60.0% |
| Gemini-2.5-Pro | 74.1% | 100.0% | 85.1% | 82.5% | 35.0% |
SafePred-8B achieves the same short-term F1 as Qwen3-8B, indicating that fine-tuning does not produce a measurable improvement in immediate-risk recognition in this evaluation. Gemini-2.5-Pro achieves a higher F1 of 85.1%, showing stronger short-term risk recognition.
(1-b) Long-term Risk Recognition.
We construct a separate set of 40 samples, comprising 20 delayed-risk cases and 20 matched safe controls base on SafePred-Bench. In each delayed-risk case, the candidate action establishes a condition that leads to an actual policy violation within at most four additional executed actions. Ground-truth outcomes are established from actual execution trajectories before prediction generation. We also compare these cases against the confirmed SafePred training and validation sets to check for task overlap and near-duplicate examples.
All models receive the same task, policies, pre-action state, candidate action, and specified continuation plan. Actual future states and outcome labels are withheld. We use each model's native prediction prompt with a unified maximum output budget of 8,192 tokens.
A prediction is counted as a risk alarm when it identifies at least one violating Policy ID. We report Precision, Recall, F1, and the false positive rate on safe controls.
| World Model | Precision ↑ | Recall ↑ | Long-term F1 ↑ | Safe False Positive Rate ↓ |
|---|---|---|---|---|
| Qwen3-8B | 58.6% | 85.0% | 69.4% | 60.0% |
| SafePred-8B | 70.0% | 70.0% | 70.0% | 30.0% |
| Gemini-2.5-Pro | 94.7% | 90.0% | 92.3% | 5.0% |
Compared with Qwen3-8B, SafePred-8B increases Precision from 58.6% to 70.0% and F1 from 69.4% to 70.0%, while reducing the false positive rate on safe controls from 60.0% to 30.0%. However, Recall decreases from 85.0% to 70.0%, indicating a trade-off between reducing false alarms and detecting all delayed risks. Gemini-2.5-Pro achieves the strongest results across the reported metrics.
W1 (continue): Heavy reliance on world-model capabilities and insufficient validation of long-term prediction reliability. The effectiveness of SafePred fundamentally depends on the underlying ability of the world model to anticipate future consequences. Although Table 7 in Appendix F compares different world-model backbones, the results reveal noticeable performance differences, while SafePred-8B achieves only modest improvements over its Qwen3-8B base model. More importantly, the evaluation primarily focuses on downstream safety and task success, without directly assessing long-term forecast accuracy across different model capacities. As discussed in Section 3.3, long-term prediction is implemented through high-level semantic reasoning. It therefore remains unclear whether smaller world models can reliably anticipate delayed risks and to what extent the improvements arise from genuine predictive capability.
Q1 (continue): Regarding world-model capability, could the authors provide a more detailed analysis of how different model capacities affect short- and long-term risk prediction accuracy? In particular, how reliably can SafePred-8B identify delayed risks compared with larger world models, beyond the downstream performance comparisons in Appendix F?
Q2 (continue): Regarding long-term risk evaluation, could the authors provide additional evidence comparing predicted long-term consequences with actual subsequent outcomes? Such an analysis would help establish whether SafePred genuinely anticipates delayed risks before they materialize.
(2) Execution-grounded Validation.
To further examine whether predicted consequences correspond to actual future outcomes, we conduct an additional analysis using 40 controlled execution fragments, including 20 delayed-risk cases and 20 matched safe controls, organized into 20 task pairs across 13 mechanism families. These fragments form a separate diagnostic set and are distinct from the 44 long-term-risk tasks in OS-Harm. We evaluate Gemini-2.5-Pro, the World Model backbone used in our main SafePred experiments.
In each delayed-risk case, the candidate action establishes a condition whose harmful consequence materializes within at most four additional executed actions. Ground-truth outcomes are established from actual file changes, database states, and execution records before prediction generation. The model receives the task, policies, pre-action state, candidate action, and specified continuation plan, while actual future states and outcome labels are withheld.
A human reviewer compares each prediction with the actual execution outcome. A full match requires correctly identifying both the affected object and its terminal outcome; semantically equivalent descriptions are accepted. Predictions identifying only intermediate authorization, permission, or queue states are classified as partial matches. All 20 delayed-risk cases are included in the evaluation.
| Assessment | Result |
|---|---|
| Full terminal-consequence match | 12/20 (60.0%) |
| Partial match | 7/20 (35.0%) |
| Non-match | 1/20 (5.0%) |
| Unsupported terminal-harm prediction on safe controls | 1/20 (5.0%) |
Overall, the human-reviewed evaluation provides direct evidence of long-term risk prediction capability. SafePred captures risk-relevant consequences in 19 of 20 delayed-risk cases (95.0%), including 12 full matches (60.0%) and 7 partial matches (35.0%), with only one missed case. The false-positive rate on safe controls is 5.0%.
We have added these prediction-level evaluations, execution-grounded results, representative cases, and limitations to the revised manuscript, with the corresponding details in Appendix F and Section 3.3.
Dear Reviewer Tn5z,
Thank you for your constructive feedback. We have carefully addressed your concerns by extending the evaluation and revising the manuscript in two aspects: (1) World Model Prediction Evaluation: We introduce additional short-term and long-term risk recognition experiments to compare Qwen3-8B, SafePred-8B, and Gemini-2.5-Pro, and further validate predicted long-term consequences against actual execution outcomes. (2) Presentation Quality: We improve the readability of Figure 4 and correct the formatting of numbered enumerations in Sections 3.1 and 4.1. The corresponding experimental results and revisions are detailed below. We hope they adequately address your concerns.