- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MMSKILLRISK, an executable safety evaluation benchmark investigating image-borne attacks embedded within multimodal agent skills. The authors propose Native-Context Visual Attack (NCVA), where malicious instructions are disguised as plausible visual elements (e.g., interface labels, notes, or annotations) within skill teaching images, while the textual documentation provides procedural guidance without explicit malicious content. Across 28 curated clean skills, 36 attack packages, and 108 executable sandbox cases spanning five risk categories, the authors evaluate nine model-harness configurations.
Critical strengths
- The paper studies a realistic and critical supply-chain vulnerability in multimodal agent skills, and moves beyond static scanner evasion to ask whether agents actually act on malicious instructions hidden in skill images.
- The evaluation setup rigorously decouples legitimate task completion from attack success (TSR vs. ASR), providing realistic evidence that an agent can fulfill user requests while silently carrying out unauthorized dangerous operations.
- The benchmark is executable and the case design is concrete. The nine configurations span three agent harnesses (Codex, Claude Code, and DeepSeek Harness) and run in isolated Docker containers with fixed resource limits and a uniform timeout, which makes the evaluation controlled and reproducible.
Critical weaknesses
- The primary limitation lies in the scale of the benchmark. With only 28 base skills, 36 attack packages, and a total of 108 evaluation instances, the coverage is quite constrained. When broken down across five risk categories, each objective contains merely 18 to 24 test cases. Such a limited sample size makes statistical conclusions somewhat fragile.
- The design and instantiation of NCVA rely entirely on manual effort by human experts to craft suitable visual elements into images. The paper does not provide an automated pipeline for crafting these instances, raising concerns about the data scalability.
- Embedding malicious instructions into skill images and referencing them via documentation has already been explored in prior work. Conceptually, NCVA is an application of established indirect visual prompt injection to agent skill settings, which the algorithmic innovation is limited.
- Although the skills include GUI scenarios, the actual execution is conducted over static reference images and files inside Docker containers rather than dynamic, multi-turn desktop environments (e.g., OSWorld). This may not fully capture the perceptual challenges and real dynamics of computer-use agents.
Questions for authors
-
Given that NCVA packages were handcrafted by human experts, have the authors explored automated methods (e.g., VLM-assisted inpainting or automatic annotation placement) to scale the benchmark beyond 108 cases?
-
Since the main outcomes are judged by gemini-3.7-flash, could you provide the human–LLM agreement on the manually reviewed subset, to support the reliability of the automatic evaluation?
-
Could the author also test a simple runtime defense rather than static scanner, such as instructing the harness to more carefully focus on the malicious prompt injection in image to show the reliability of NCVA under defense scenario.
2 comments
W4: Static Inputs and Execution Risks
Thank you for raising this concern. Our evaluation mainly focuses on how static visual resources in multimodal skills influence agents' actual actions. Although the task screenshots are fixed, agents still read skill instructions, inspect images, invoke tools, and produce artifacts; the visual resources form an important part of this execution process. Our experiments also show that injected instructions can induce concrete operations, including file modifications, configuration changes, and data transfers to a local test receiver. Moreover, our benchmark includes multimodal skills sourced or adapted from public skill ecosystems and prior research. Therefore, our evaluation setting with static tasks and images also has practical relevance.
Extending the evaluation to dynamic GUI environments, including visual localization, changing interface states, and continuous desktop interaction, is an important direction for our future work. Appendix H already acknowledges the current evaluation scope. We have also added Appendix E.7, Static Visual Inputs and Agent Execution, to clarify our focus and the distinction between static visual inputs and actual agent execution.
Q2: Human–LLM Agreement
Thank you for asking about human–LLM agreement. We have expanded Appendix D.1 to provide details of the manual validation, including the sample size, sampling procedure, agreement results, and disagreement analysis. Specifically, for each of the original nine model–harness configurations, we randomly sampled two cases from each of the five attack categories, resulting in 90 reviewed runs. Human reviewers evaluated attack success and legitimate-task completion based on execution traces, task artifacts, and predefined scoring criteria. Their initial assessments matched the LLM judge on both outcomes in 89 of 90 runs, giving a run-level exact agreement rate of 98.9%. In the remaining run, the human assessment differed from the LLM judge on one criterion. We carefully revisited the execution evidence under the predefined criteria and confirmed that the judge's original decision was correct, so the score was retained. Appendix D.1 now documents this case and how the disagreement was resolved.
Q3: Runtime Defense
Thank you for this suggestion. Guardrails are a common form of runtime defense in agent security. We have added an exploratory experiment in Appendix E.5 of the revised manuscript. Specifically, we evaluate a lightweight runtime guardrail on 30 cases using Kimi-K3 with Claude Code. The guardrail uses Claude Code's PreToolUse hook and deny rules to inspect tool calls before execution and restrict selected high-risk operations. With the guardrail enabled, ASR decreases from 76.7% to 46.7%, while legitimate-task success drops from 100.0% to 73.3%. These results show that this defense configuration reduces malicious operation submissions but also substantially affects legitimate-task completion, illustrating the trade-off between security protection and task completion.
We sincerely thank you for your careful review and thoughtful suggestions. Your feedback has helped us strengthen the manuscript and clarify its presentation. We have revised the paper accordingly and highlighted all changes in red for your convenience. Below, we respond to your comments point by point. We appreciate the time and effort you have devoted to evaluating our work.
W1 & W2 & Q1: Benchmark Scale and Construction Scalability
Thank you for highlighting the limitations in benchmark scale and construction scalability. We agree that the original benchmark was limited in size. For reference, Skill-Inject [1], SkillSafetyBench [2], and OpenSkillRisk [3] contain 202, 155, and 263 executable cases, respectively, compared with our original 108 cases. In the revision, we added 16 base skills and 99 executable cases, expanding the benchmark to 44 base skills, 69 attack packages, and 207 cases. The number of cases per attack objective increased from approximately 18–24 to 39–42. The additions include 10 public multimodal skills from AutoVisualSkill and six skills from MMSkills. The expanded experimental results and category-level analyses are reported in the revised manuscript.
Regarding automated construction, we would first like to clarify that NCVA does not rely entirely on manual work. As stated in Section 4.2 of the original submission, human experts design the case-specific mechanisms, while agents assist with implementation. We did explore fully automated attack construction using VLMs. However, most current LLMs are safety-aligned and refuse certain requests to construct and embed malicious payloads. In our exploration, models that complied with these requests did not consistently produce results meeting our requirements for workflow alignment and visual integration. We therefore adopted a human–agent collaborative process, with agents assisting implementation and humans handling key design decisions and quality review. This process requires substantial human effort but allows better control over case quality. We have expanded Appendices C.1 and C.2 to describe the construction process and division of responsibilities. We hope these clarifications can address your concerns.
[1] Schmotz et al. (2026). Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. arXiv:2602.20156v3
[2] Jin et al. (2026). SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces.
[3] Liu et al. (2026). OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills. arXiv:2607.20121v2
W3: Relationship to Prior Work and Contributions
Thank you for pointing out the connection to prior work. SkillCamo has indeed explored hiding malicious instructions in skill images, and we discussed this connection in the related work of the original submission. The main differences between the two works are as follows:
Our main contribution is an executable benchmark for end-to-end safety evaluation of multimodal skills, systematically examining how malicious instructions in images affect agents' actual behavior. SkillCamo's evaluation primarily focuses on pre-deployment skill screening, and its attack success rate measures how often malicious skills bypass scanners. It does not report an end-to-end evaluation on actual tasks comparable to ours. We run agents on tasks, use execution traces to determine whether they carry out malicious operations, and independently evaluate legitimate-task completion.
The multimodal skills we select and construct already use teaching images as visual references in their normal workflows. Before any attack is introduced, these images contain information needed to complete legitimate tasks.
SkillCamo renders malicious commands or their fragments as newly added image resources and modifies the skill documentation to introduce them into the workflow. NCVA embeds malicious instructions into teaching images that already serve legitimate purposes in clean skills, using elements such as chart annotations, interface labels, and workflow descriptions. It preserves the visual information required for legitimate tasks while integrating the attack content into the instructional scene.