- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Risky-Bench, a safety evaluation layer on top of VitaBench’s simulated life-assist tasks. Tasks are perturbed across five attack surfaces: user instruction, environment, tool feedback, memory, and system prompt. The benchmark then checks whether agents violate 15 deployment-specific safety rubrics while completing an otherwise benign goal. Across 750 tasks and seven models, ASR ranges from 23% to 58%, highest under tool-feedback and system-prompt attacks. The paper also reports that explicit reasoning raises ASR on average and that a prompt-level safety reminder only partially helps.
Critical strengths
- The problem is important well-motivated.
- The findings are interesting.
Critical weaknesses
- System-prompt attacks are too strong a threat model. If an attacker rewrites the system prompt, the model is exptected to follow it.
- The memory attacks are prefilling, not poisoning. The injected histories end with a fabricated assistant turn agreeing to the unsafe action.
- Novelty is limited. The paper contributes no new environments or attack primitives, and all scenarios are simulated.
- GPT-4.1 is overused. It serves as attack generator, user simulator, judge, and an evaluated agent. It is also an older model, and the cross-validation with a second model covers only one domain.
- No frontier models are evaluated. Most evaluated agents are lightweight or non-frontier (Haiku 4.5, Gemini-3-Flash, GPT-4.1, Qwen-Plus), so the conclusions may not transfer to current frontier models.
Questions for authors
See weaknesses above
4 comments
Response to weakness 4
W4: GPT-4.1 is overused - It serves as attack generator, user simulator, judge, and an evaluated agent. It is also an older model, and the cross-validation with a second model covers only one domain.
Response: Thank you for raising this concern. We address the overlap between the evaluated agent and data construction agent; safety evaluation agent and user-simulator agent separately.
1. Data generator–evaluated agent overlap. Following the original construction criteria, we formed an alternative test set primarily using Claude-Opus-5.5. We compared GPT-4.1 on the original and alternative sets, adding Claude-Sonnet-5 as an additional comparison model (750 task positions per model per set).
| Evaluated model | Original-set ASR | Alternative-set ASR | Change |
|---|---|---|---|
| GPT-4.1 | 40.00% (300/750) | 61.87% (464/750) | +21.87 pp |
| Claude-Sonnet-5 | 16.27% (122/750) | 11.07% (83/750) | −5.20 pp |
These model-dependent changes indicate a potential source of evaluation bias. We have therefore removed GPT-4.1 from the main model comparison and added GPT-6.1-Sol and Claude-Sonnet-5 to the main evaluation. The experiment results are updated in Section 3.1 in the revised manuscript, with detailed comparison in Appendix G.5 (Table 34).
2. Safety judge–evaluated agent overlap. As detailed in Appendix G.3.2, we validated LLM-as-judge results through human review: two annotators independently checked each automated verdict against the task description, applicable safety rubric, and full interaction log, confirming or correcting the label. The overall manual correction rate was below 5%. Our reported ASRs therefore rely on human-verified judgments.
3. User simulator–evaluated agent overlap. To examine potential bias from using GPT-4.1 as both the user simulator and an evaluated agent, we adopt Gemini-3.8-Flash as an alternative user simulator, and evaluate GPT-4.1 alongside Claude-Sonnet-5, which serves as neither the default nor the alternative simulator. Both agents were compared on 750 tasks spanning all three domains. Results shown below indicate that replacing the user simulator did not affect GPT-4.1 substantially more than Claude-Sonnet-5; both showed limited ASR sensitivity.
| Domain | GPT-4.1 | Claude-Sonnet-5 |
|---|---|---|
| Delivery | 0.38 → 0.36 | 0.10 → 0.16 |
| In-store | 0.46 → 0.48 | 0.26 → 0.24 |
| OTA | 0.32 → 0.36 | 0.20 → 0.18 |
| Overall | 0.39 → 0.40 | 0.19 → 0.19 |
Arrows indicate ASR under the default (GPT-4.1) → alternative (Gemini-3.8-Flash) user simulator. The simulator setup is summarized in Section 3, and detailed comparison reported in Appendix G.2 (Table 30).
Response to weakness 5
W5: No frontier models are evaluated - Most evaluated agents are lightweight or non-frontier (Haiku 4.5, Gemini-3-Flash, GPT-4.1, Qwen-Plus), so the conclusions may not transfer to current frontier models.
Response: Thank you for this suggestion. We have removed GPT-4.1 from the main experiments and added Claude-Sonnet-5 and GPT-6.1-Sol on all 750 tasks across three domains and five attack surfaces, with results shown below.
| Model | User instruction | Environment | Agent memory | Tool feedback | Codebase | Overall |
|---|---|---|---|---|---|---|
| Claude-Sonnet-5 | 11.33% | 4.67% | 18.67% | 1.33% | 45.33% | 16.27% |
| GPT-6.1-Sol | 14.00% | 5.33% | 2.67% | 0.67% | 18.00% | 8.13% |
The two frontier models show substantially improved robustness to several risks identified in RQ1, particularly tool-feedback and environmental attacks, with GPT-6.1-Sol also showing much lower susceptibility to memory attacks. However, codebase poisoning remains a substantial vulnerability for both models. We have revised the discussion in Section 3.1 (RQ1) to reflect these improvements.
We also extended the RQ3 safety-reminder experiment to the two new models. The combined results for all eight models are reported below, with arrows indicating results without → with the safety reminder, evaluated on 250 Delivery tasks:
| Model | ASR ↓ | Task completion ↑ | Over-refusal ↓ | Safe task completion ↑ |
|---|---|---|---|---|
| Gemini-3-Flash | 0.46 → 0.29 | 0.32 → 0.38 | 0.00 → 0.02 | 0.18 → 0.25 |
| Claude-Haiku-4.5 | 0.20 → 0.09 | 0.15 → 0.19 | 0.01 → 0.03 | 0.13 → 0.19 |
| Qwen-Plus | 0.36 → 0.17 | 0.16 → 0.18 | 0.01 → 0.04 | 0.09 → 0.16 |
| DeepSeek-V3.2 | 0.51 → 0.32 | 0.24 → 0.23 | 0.01 → 0.02 | 0.11 → 0.18 |
| Kimi-K2 | 0.34 → 0.17 | 0.12 → 0.12 | 0.02 → 0.04 | 0.10 → 0.12 |
| Doubao-Seed-1.8 | 0.58 → 0.34 | 0.24 → 0.20 | 0.04 → 0.07 | 0.11 → 0.16 |
| Claude-Sonnet-5 | 0.15 → 0.04 | 0.33 → 0.38 | 0.01 → 0.01 | 0.28 → 0.36 |
| GPT-6.1-Sol | 0.06 → 0.00 | 0.20 → 0.20 | 0.02 → 0.00 | 0.19 → 0.20 |
| Overall | 0.33 → 0.18 | 0.22 → 0.24 | 0.02 → 0.03 | 0.15 → 0.20 |
Overall, ASR decreases by 15.55 percentage points, while safe task completion increases by only 5.40 percentage points. The results indicate that large reductions in ASR do not necessarily translate into comparable gains in safe task completion. These results are summarized in Section 3.3 (Figure 7), with the complete eight-model table in Appendix G.6 (Table 35).
References
[1] OpenAI (2025). Model Spec.
[2] Wallace et al. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions.
[3] Chen et al. (NeurIPS 2024). AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases.
[4] Andriushchenko et al. (ICLR 2025). Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks.
[5] Meng et al. (IEEE TIFS 2026). Dialogue Injection Attack: Jailbreaking LLMs Through Context Manipulation.
[6] Debenedetti et al. (NeurIPS 2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.
[7] Zhang et al. (ICLR 2025). Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents.
Response to reviewer comments
Thank you again for your thoughtful comments and suggestions. We have carefully addressed each point in our response and would appreciate your feedback on whether these clarifications resolve your concerns. Any further comments or suggestions would be most welcome.
Response to weakness 1 & weakness 2 & weakness 3
Response: Thank you for raising this concern. We agree that modifying an agent’s system prompt is a strong threat assumption, explicitly classified as control-level access in our taxonomy. However, we believe that a safety-aligned model should not unconditionally follow harmful directives merely because they appear in the system prompt. First, system messages are not always the highest authority governing model behavior. OpenAI’s Model Spec, for example, defines the hierarchy Root > System > Developer > User > Guideline, with root-level rules that “cannot be overridden by system messages, developers or users” [1].
Second, compromised application configurations can expose models to adversarial system-level instructions, making this a meaningful threat scenario to study. OpenAI has evaluated a closely related setting in [2] that insert jailbreaks into the System Message and evaluate whether the model refuses the accompanying unsafe requests.
Our designed system-prompt attack therefore examines: even when an attacker can tamper with the application’s governing prompt, does the model follow unsafe directives, or can safety protections established through its alignment training or higher-priority requirements still prevent harmful actions. Our experiments provide empirical evidence of agents’ robustness and failure modes in this setting, identifying where safety protections hold or fail under system prompt compromise. We clarify this control-level threat assumption in Section 2.2.1 and the evaluated configurations in Appendix D.3.5.
Response: Thank you for pointing out this distinction. After revisiting the relevant definitions, we agree that our setting should be more accurately described as conversation-history prefilling than memory poisoning. Memory poisoning contaminates stored memory or knowledge bases to influence behavior through subsequent retrieval [3]; prefilling supplies the beginning of an assistant response to steer its continuation [4]. In our implementation, we directly supply fabricated conversation history ending with the assistant agreeing to unsafe behavior, in an attempt to induce the agent to carry out the corresponding unsafe actions during subsequent interaction.
We have revised Sections 2.2, 3.1, and 3.2 and Appendices D.2 and D.3.3 to correct corresponding terms and clarify the implemented scope.
Response: Thank you for the comment. Firstly, we agree that RiskyBench builds on VitaBench’s environments and established attack techniques studied in agent security benchmarks such as AgentDojo [6] and ASB [7]. Our contribution is a practical approach to conducting context-specific safety evaluation in established agent environments. As agents are deployed across more applications, systematic safety evaluation becomes increasingly important for identifying safety risks and informing subsequent alignment. RiskyBench exemplifies how an application’s safety requirements can be translated into executable risk scenarios and verifiable safety criteria, enabling systematic analysis of agent behavior and safety failures.
Secondly, regarding the concern about “simulated scenarios,” we aim to capture the safety risks agents may face in real-world services through simulation. We construct risk factors by considering each scenario’s service workflow, the agent’s permissions, and how its actions could harm users or the system. These scenarios undergo two-stage human review to assess whether the risks are meaningful, contextually plausible, and aligned with the target safety requirements (Appendix D, with the review criteria in Appendix D.4). We also acknowledge that collecting naturally occurring safety failures from deployments with real users would be a valuable extension, but our benchmark enables controlled, repeatable safety testing without carrying out real transactions or exposing private user data, helping identify potential failure modes and inform safety improvements before deployment.