acceptodds
Withdrawn from ICLR 2027

MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent skills are shareable packages of procedural instructions, tools, and examples. Multimodal skills additionally include visual references that agents retrieve and inspect during execution. Because these images guide actions, attackers can disguise malicious instructions as ordinary visual guidance within otherwise legitimate skills. Existing skill-security research primarily examines text-carried attacks or scanner detection, leaving the runtime effects of image-borne attacks insufficiently evaluated. We introduce MMSkillRisk, to our knowledge the first publicly available benchmark dedicated to end-to-end safety evaluation of image-borne attacks in multimodal skills. To instantiate this attack surface, we design Native-Context Visual Attack (NCVA), which disguises malicious instructions as native components of teaching images, such as annotations and interface labels. The accompanying SKILL.md provides auxiliary guidance toward relevant visual regions without explicitly stating the malicious operation. Built from 28 curated clean skills, MMSkillRisk contains 36 attack packages and 108 executable cases spanning five attack objectives, with separate checks for attack success and legitimate-task completion. Across nine model–harness configurations evaluated in isolated sandboxes, NCVA induces unauthorized operations in every configuration. Its pooled attack success rate (ASR) reaches 43.1%, exceeding the matched text-carrier baseline by 16.4 percentage points, with higher ASR in all nine configurations. Attack success and legitimate-task completion co-occur in 36.5% of cases, reaching 72.2% for GPT-5.6-sol with Codex. These results show that skill-bundled images can induce unauthorized actions even as agents complete legitimate tasks, so task success alone does not establish safe skill use. Anonymized code and data are available at [https://anonymous.4open.science/r/MMSkillRisk_-F36F/](https://anonymous.4open.science/r/MMSkillRisk_-F36F/).

Primary area:
datasets and benchmarks
Keywords:
Multimodal Agents, Agent Skills, Agent Safety, AI Security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G93BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MMSKILLRISK, an executable safety evaluation benchmark investigating image-borne attacks embedded within multimodal agent skills. The authors propose Native-Context Visual Attack (NCVA), where malicious instructions are disguised as plausible visual elements (e.g., interface labels, notes, or annotations) within skill teaching images, while the textual documentation provides procedural guidance without explicit malicious content. Across 28 curated clean skills, 36 attack packages, and 108 executable sandbox cases spanning five risk categories, the authors evaluate nine model-harness configurations.

Critical strengths
  1. The paper studies a realistic and critical supply-chain vulnerability in multimodal agent skills, and moves beyond static scanner evasion to ask whether agents actually act on malicious instructions hidden in skill images.
  2. The evaluation setup rigorously decouples legitimate task completion from attack success (TSR vs. ASR), providing realistic evidence that an agent can fulfill user requests while silently carrying out unauthorized dangerous operations.
  3. The benchmark is executable and the case design is concrete. The nine configurations span three agent harnesses (Codex, Claude Code, and DeepSeek Harness) and run in isolated Docker containers with fixed resource limits and a uniform timeout, which makes the evaluation controlled and reproducible.
Critical weaknesses
  1. The primary limitation lies in the scale of the benchmark. With only 28 base skills, 36 attack packages, and a total of 108 evaluation instances, the coverage is quite constrained. When broken down across five risk categories, each objective contains merely 18 to 24 test cases. Such a limited sample size makes statistical conclusions somewhat fragile.
  2. The design and instantiation of NCVA rely entirely on manual effort by human experts to craft suitable visual elements into images. The paper does not provide an automated pipeline for crafting these instances, raising concerns about the data scalability.
  3.  Embedding malicious instructions into skill images and referencing them via documentation has already been explored in prior work. Conceptually, NCVA is an application of established indirect visual prompt injection to agent skill settings, which the algorithmic innovation is limited.
  4. Although the skills include GUI scenarios, the actual execution is conducted over static reference images and files inside Docker containers rather than dynamic, multi-turn desktop environments (e.g., OSWorld). This may not fully capture the perceptual challenges and real dynamics of computer-use agents.
Questions for authors
  1. Given that NCVA packages were handcrafted by human experts, have the authors explored automated methods (e.g., VLM-assisted inpainting or automatic annotation placement) to scale the benchmark beyond 108 cases?

  2. Since the main outcomes are judged by gemini-3.7-flash, could you provide the human–LLM agreement on the manually reviewed subset, to support the reliability of the automatic evaluation?

  3. Could the author also test a simple runtime defense rather than static scanner, such as instructing the harness to more carefully focus on the malicious prompt injection in image to show the reliability of NCVA under defense scenario.

2 comments

We sincerely thank you for your careful review and thoughtful suggestions. Your feedback has helped us strengthen the manuscript and clarify its presentation. We have revised the paper accordingly and highlighted all changes in red for your convenience. Below, we respond to your comments point by point. We appreciate the time and effort you have devoted to evaluating our work.

W1 & W2 & Q1: Benchmark Scale and Construction Scalability

Thank you for highlighting the limitations in benchmark scale and construction scalability. We agree that the original benchmark was limited in size. For reference, Skill-Inject [1], SkillSafetyBench [2], and OpenSkillRisk [3] contain 202, 155, and 263 executable cases, respectively, compared with our original 108 cases. In the revision, we added 16 base skills and 99 executable cases, expanding the benchmark to 44 base skills, 69 attack packages, and 207 cases. The number of cases per attack objective increased from approximately 18–24 to 39–42. The additions include 10 public multimodal skills from AutoVisualSkill and six skills from MMSkills. The expanded experimental results and category-level analyses are reported in the revised manuscript.

Regarding automated construction, we would first like to clarify that NCVA does not rely entirely on manual work. As stated in Section 4.2 of the original submission, human experts design the case-specific mechanisms, while agents assist with implementation. We did explore fully automated attack construction using VLMs. However, most current LLMs are safety-aligned and refuse certain requests to construct and embed malicious payloads. In our exploration, models that complied with these requests did not consistently produce results meeting our requirements for workflow alignment and visual integration. We therefore adopted a human–agent collaborative process, with agents assisting implementation and humans handling key design decisions and quality review. This process requires substantial human effort but allows better control over case quality. We have expanded Appendices C.1 and C.2 to describe the construction process and division of responsibilities. We hope these clarifications can address your concerns.

[1] Schmotz et al. (2026). Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. arXiv:2602.20156v3

[2] Jin et al. (2026). SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces.

[3] Liu et al. (2026). OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills. arXiv:2607.20121v2

W3: Relationship to Prior Work and Contributions

Thank you for pointing out the connection to prior work. SkillCamo has indeed explored hiding malicious instructions in skill images, and we discussed this connection in the related work of the original submission. The main differences between the two works are as follows:

  1. Our main contribution is an executable benchmark for end-to-end safety evaluation of multimodal skills, systematically examining how malicious instructions in images affect agents' actual behavior. SkillCamo's evaluation primarily focuses on pre-deployment skill screening, and its attack success rate measures how often malicious skills bypass scanners. It does not report an end-to-end evaluation on actual tasks comparable to ours. We run agents on tasks, use execution traces to determine whether they carry out malicious operations, and independently evaluate legitimate-task completion.

  2. The multimodal skills we select and construct already use teaching images as visual references in their normal workflows. Before any attack is introduced, these images contain information needed to complete legitimate tasks.

  3. SkillCamo renders malicious commands or their fragments as newly added image resources and modifies the skill documentation to introduce them into the workflow. NCVA embeds malicious instructions into teaching images that already serve legitimate purposes in clean skills, using elements such as chart annotations, interface labels, and workflow descriptions. It preserves the visual information required for legitimate tasks while integrating the attack content into the instructional scene.

W4: Static Inputs and Execution Risks

Thank you for raising this concern. Our evaluation mainly focuses on how static visual resources in multimodal skills influence agents' actual actions. Although the task screenshots are fixed, agents still read skill instructions, inspect images, invoke tools, and produce artifacts; the visual resources form an important part of this execution process. Our experiments also show that injected instructions can induce concrete operations, including file modifications, configuration changes, and data transfers to a local test receiver. Moreover, our benchmark includes multimodal skills sourced or adapted from public skill ecosystems and prior research. Therefore, our evaluation setting with static tasks and images also has practical relevance.

Extending the evaluation to dynamic GUI environments, including visual localization, changing interface states, and continuous desktop interaction, is an important direction for our future work. Appendix H already acknowledges the current evaluation scope. We have also added Appendix E.7, Static Visual Inputs and Agent Execution, to clarify our focus and the distinction between static visual inputs and actual agent execution.

Q2: Human–LLM Agreement

Thank you for asking about human–LLM agreement. We have expanded Appendix D.1 to provide details of the manual validation, including the sample size, sampling procedure, agreement results, and disagreement analysis. Specifically, for each of the original nine model–harness configurations, we randomly sampled two cases from each of the five attack categories, resulting in 90 reviewed runs. Human reviewers evaluated attack success and legitimate-task completion based on execution traces, task artifacts, and predefined scoring criteria. Their initial assessments matched the LLM judge on both outcomes in 89 of 90 runs, giving a run-level exact agreement rate of 98.9%. In the remaining run, the human assessment differed from the LLM judge on one criterion. We carefully revisited the execution evidence under the predefined criteria and confirmed that the judge's original decision was correct, so the score was retained. Appendix D.1 now documents this case and how the disagreement was resolved.

Q3: Runtime Defense

Thank you for this suggestion. Guardrails are a common form of runtime defense in agent security. We have added an exploratory experiment in Appendix E.5 of the revised manuscript. Specifically, we evaluate a lightweight runtime guardrail on 30 cases using Kimi-K3 with Claude Code. The guardrail uses Claude Code's PreToolUse hook and deny rules to inspect tool calls before execution and restrict selected high-risk operations. With the guardrail enabled, ASR decreases from 76.7% to 46.7%, while legitimate-task success drops from 100.0% to 73.3%. These results show that this defense configuration reduces malicious operation submissions but also substantially affects legitimate-task completion, illustrating the trade-off between security protection and task completion.

Reviewer_9PVwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates image-borne attacks in multimodal agent skills and proposes Native-Context Visual Attack (NCVA), which embeds malicious instructions into seemingly benign instructional images, using SKILL.md to guide agents toward the relevant regions without explicitly stating the malicious actions. The authors introduce MMSKILLRISK, a benchmark of 108 executable cases covering five attack objectives. Across nine model–harness configurations, NCVA achieves a 43.1% attack success rate (ASR), outperforming the text-only baseline by 16.4 percentage points. Attack success co-occurs with task completion in 36.5% of cases, showing that task completion does not guarantee safety. Ablations confirm the complementary roles of images and textual guidance, while five existing skill scanners detect fewer than 12% of poisoned packages, exposing a significant gap in current defenses.

Critical strengths
  1. Accurate positioning against related work. The paper clearly distinguishes itself from the closest related work, SkillCamo (Jia et al., 2026b). While the latter focuses on whether malicious content is detected by scanners, this paper investigates whether agents actually follow the malicious instructions and perform unauthorized actions during real execution. This distinction between runtime exploitation and detection is valuable.

  2. Separately evaluating attack success and task completion is a methodological strength. The TC-ASR metric effectively captures an underestimated risk: an agent can complete the user's task perfectly while simultaneously executing the attacker's instructions. Therefore, task success rates alone are insufficient to assess security.

  3. Strong end-to-end engineering and evaluation. The paper combines executable end-to-end attacks, isolated sandbox environments, and an analysis of defenses provided by real-world skill scanners, demonstrating a high level of engineering completeness.

Critical weaknesses
  1. Permission settings undermine practical validity. All harnesses run with sandbox restrictions bypassed, approvals skipped, and danger-full-access enabled, yet this is dismissed in one sentence ("characterizes configured systems, not default interactive settings"). In real deployments, these permission gates would block many of the examined operations, so the headline 43.1% ASR likely overestimates the actual threat. The lack of a comparison under default interactive permissions is the paper's most significant practical weakness.

  2. Small benchmark, mostly author-constructed. Of 28 skills, only 5 are publicly collected and used as-is; the rest are 8 MMSkills adaptations, 10 custom GUI skills, and 5 custom visual workflows. More than half the attack surface is manually constructed, with no evidence that these custom workflows and visuals are representative of real skill ecosystems—making the claim of real-world ecological validity hard to substantiate.

  3. Tasks are static, lacking realistic interaction. The MMSkills adaptations are described as "offline state-reading tasks that do not reproduce the original OSWorld interactions." The benchmark essentially evaluates static image inspection rather than dynamic, multi-step agent behavior on real desktops/GUIs. Whether the attacks survive dynamic visual localization and changing interface states is unknown. The authors acknowledge this, but it directly limits generalizability.

  4. LLM judge reliability is insufficiently validated. All ASR/TSR results come from gemini-3.7-flash. Appendix D.1 mentions a manually reviewed "subset" but reports no sample size, no inter-rater agreement (e.g., Cohen's κ), and no analysis of disagreements. For a benchmark paper selling measurability, resting all results on an unvalidated judge is a critical gap.

  5. Weak statistical rigor. No indication of repeated runs, no confidence intervals, no significance tests, and sampling temperatures are explicitly not standardized. In Figure 3, each category-specific ASR rests on only ~18–24 samples, so it is hard to tell how much of the radar-chart variation is genuine signal versus noise.

  6. Questionable text-baseline fairness. The baseline writes the full payload as plaintext in SKILL.md—precisely the form scanners catch most easily and safety training most likely targets. The 16.4-point gap may therefore conflate modality differences with differential safety training against explicit text injection, rather than showing images are inherently more dangerous. A text baseline disguising the payload as ordinary workflow steps would be a fairer, stronger control.

  7. Attack-only; no defensive baseline. The paper stops at showing existing scanners fail, without testing even a simple mitigation (e.g., VLM pre-screening of images, or OCR-ing image text into the scanning pipeline). This weakens the constructive value of the contribution.

Questions for authors

refer to weaknesses

3 comments

Thank you very much for your insightful and constructive comments. We sincerely appreciate your detailed feedback that helps us further improve the quality and clarity of the manuscript. We have carefully revised the paper according to your valuable suggestions. For your convenience, all changes are highlighted in red in the revised manuscript. Below, we provide our point-by-point responses to each of your comments. Thank you again for your time and effort in reviewing our work.

W1: Permission Settings

Thank you for raising this concern. Our original explanation in the appendix was indeed too brief to clearly communicate the purpose and limitations of this setting. We use this configuration to assess whether agents can independently recognize and reject hidden risky instructions in skills, without relying on external intervention such as human approval for individual operations. This choice follows the non-interactive evaluation practices in the public implementations of representative skill benchmarks, including Skill-Inject [1], OpenSkillRisk [2], SkillSafetyBench [3], and Skill-Use [4]. We acknowledge that default interactive permissions may indeed prevent some malicious operations from taking effect. However, outcomes under those settings also depend on external approval and access-control decisions, beyond the agent's own ability to identify and reject malicious instructions. Our reported ASR therefore characterizes agent behavior under the specified non-interactive configuration and should not be interpreted as the rate of realized harm under default deployment settings. We have added a clearer and more detailed explanation of this rationale and its scope in Appendix E.4, Permission Settings: Rationale and Scope.

[1] Schmotz et al. (2026). Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. arXiv:2602.20156v3

[2] Liu et al. (2026). OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills. arXiv:2607.20121v2

[3] Jin et al. (2026). SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces. arXiv:2605.12015v1

[4] Han et al. (2026). Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses? arXiv:2608.04828v1

W2: Benchmark Coverage

Thank you for highlighting the limitations in skill sources and coverage. We agree that the original benchmark was relatively small. For context, Skill-Inject, SkillSafetyBench, and OpenSkillRisk contain 202, 155, and 263 executable cases, respectively, compared with 108 in our original version. In the revision, we have added 16 base skills and 99 executable cases, expanding the benchmark to 44 base skills, 69 malicious variants, and 207 cases. Ten of the added skills are based on publicly released multimodal skills from AutoVisualSkill, while the other six come from publicly released multimodal skills from MMSkills. These additions broaden the coverage of skill sources and workflows. Results from the expanded benchmark are reported in the revised manuscript.

Besides, we would like to clarify that our goal of this work is to test whether visual instructions embedded in multimodal skills can induce agents to perform unauthorized operations. We do not claim in either the main text or the appendix that the constructed scenarios represent the entire real-world skill ecosystem. Indeed, Appendix H explicitly acknowledges that these cases cannot represent the full range of usage scenarios or the risk distribution of real skill ecosystems.

W3: Static Inputs and Execution Risks

Thank you for raising this concern. Our evaluation mainly focuses on how static visual resources in multimodal skills influence agents' actual actions. Although the task screenshots are fixed, agents still read skill instructions, inspect images, invoke tools, and produce artifacts; the visual resources form an important part of this execution process. Our experiments also show that injected instructions can induce concrete operations, including file modifications, configuration changes, and data transfers to a local test receiver. Moreover, our benchmark includes multimodal skills sourced or adapted from public skill ecosystems and prior research. Therefore, our evaluation setting with static tasks and images also has practical relevance.

Besides, extending the evaluation to dynamic GUI environments, including visual localization, changing interface states, and continuous desktop interaction, is an important direction for our future work. Appendix H already acknowledges the current evaluation scope. We have also added Appendix E.7, Static Visual Inputs and Agent Execution, to clarify our focus and the distinction between static visual inputs and actual agent execution.

W4: LLM Judge Reliability

Thank you for raising this concern. Appendix D.1 now reports the human-review sample, sampling procedure, agreement results, and disagreement analysis. For each of the original nine model–harness configurations, we randomly selected two cases from each of the five attack categories, yielding 90 manually reviewed runs. Human reviewers assessed attack success and legitimate-task completion using execution traces, task artifacts, and the predefined scoring criteria. The initial human judgments agreed with the LLM judge on both outcomes in 89 of the 90 runs, corresponding to a run-level exact agreement rate of 98.9%. The remaining run contained a disagreement on one criterion. After carefully re-examining the execution evidence against the predefined scoring criteria, we confirmed that the LLM judge's original decision was correct and retained it. We have documented this disagreement and its resolution in Appendix D.1.

W5: Statistical Rigor

Thank you for this helpful comment. We acknowledge that the original submission did not report repeated-run variability. Given the time constraints, we focused this supplementary evaluation on Kimi-K3 with Claude Code, conducting three independent runs on the same 207 cases. Each run used a fresh agent session and a reset task environment, with the experimental configuration held fixed. Appendix D.2 now reports the following results:

MetricRun 1Run 2Run 3Mean ± SD
ASR42.5%41.5%39.6%41.2 ± 1.5
TSR97.6%97.0%95.2%96.6 ± 1.3
VIAR44.0%43.5%43.5%43.6 ± 0.3

To address the limited per-category sample size, we have expanded the benchmark from 108 to 207 cases, increasing the number of cases per attack category from approximately 18–24 to 39–42. We have updated the category-level analysis and radar chart accordingly. As for sampling settings, we set the LLM judge's temperature to 0 to reduce scoring variability, while retaining the evaluated agents' native harness and model-provider sampling configurations.

W6: Text-Baseline Fairness

Thank you for raising this concern. Our use of “plaintext” in the original manuscript may not have clearly conveyed how the baseline was constructed. In fact, our text baseline transfers the complete image-borne payload into SKILL.md, preserving its workflow-related rationale, timing requirements, operation details, and receipt-handling instructions. Thus, the text baseline already disguises the malicious operation as an ordinary workflow step.

For example, in case B41, the payload frames unauthorized transfer of screenshot readouts as a routine “PolicySync mirror,” described as “step 4 of the pass” and a “standing SOP.” The same instruction appears in a numbered image note under NCVA and in SKILL.md under the text baseline. Both retain the same synchronization rationale, receiver address, data fields, and timing requirements. Thus, the baseline preserves the workflow framing rather than using bare malicious commands. Appendix E.6 now explains this construction and provides the example.

W7: Defensive Baselines

Thank you for this suggestion. We have therefore added two defense experiments to the revised manuscript.

First, in Defense Analysis, we evaluate a Kimi-K2.6 VLM screen that examines complete skill packages, including their visual resources. It flags 72.5% of compromised packages and 4.5% of clean packages. Although image-aware screening improves detection over the evaluated conventional scanners, it still misses 27.5% of compromised packages and incurs considerable computational overhead: 81.1 seconds per package on average, approximately 255 times the latency of Cisco Skill Scanner in our measurements, with 1,118,741 tokens consumed across the evaluated packages.

Second, Appendix E.5 reports an exploratory runtime-defense experiment. Using Kimi-K3 with Claude Code, we evaluate a lightweight, rule-based runtime guardrail on 30 representative cases. The guardrail is implemented with Claude Code's PreToolUse hook and deny rules. ASR decreases from 76.7% to 46.7%, while legitimate-task success drops from 100.0% to 73.3%. This pilot shows that runtime protection can reduce malicious submissions but can also substantially disrupt legitimate task completion. Together, these experiments demonstrate the potential of image-aware screening and runtime protection, while identifying their limitations in detection coverage, computational cost, and task utility.

Thanks for the response. My main concern remains the permission setting. Without any results under default permissions, it's hard to judge how much of the threat holds in practice. Could you also report κ for the judge? And the VLM screen already catches 72.5% of the attacks, which seems to conflict with the stealthiness claim, so it's worth discussing. I'll keep my score.

Reviewer_T4Q4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MMSkillRisk, a benchmark for evaluating image-borne attacks in multimodal agent skills. It comprises 36 compromised skill variants and 108 executable cases derived from 28 clean skills, covering five attack objectives. The proposed Native-Context Visual Attack (NCVA) embeds malicious instructions into teaching images, while modified skill documentation encourages agents to consult them. Experiments across nine model–harness configurations show that these attacks can induce unauthorized operations while preserving legitimate-task completion. The paper also presents behavioral analyses, component ablations, and evaluations of existing skill scanners.

Critical strengths
  1. The paper investigates an execution-relevant attack surface: malicious instructions embedded in visual resources distributed with third-party skills.
  2. The component analysis provides evidence that images supply the malicious instructions while textual guidance increases the likelihood that agents treat them as actionable procedures.
  3. Testing nine model–harness combinations demonstrates that the vulnerability extends beyond a single system and highlights the influence of the surrounding agent framework.
Critical weaknesses
  1. Insufficient evidence of attack stealthiness. The paper describes matching the payload’s typography, layout, and semantic context to the original image, but does not systematically evaluate whether the resulting instructions actually appear benign. Without measuring detectability, the practical relevance of the claimed concealment remains unclear. Low detection rates from scanners focused primarily on text and code do not resolve this concern.
  2. NCVA offers a general design principle, but the actual attacks rely on experts constructing case-specific mechanisms. The paper does not provide a sufficiently specified, unified procedure for generating attacks across diverse skills or demonstrate generalization to unseen workflows.
  3. Limited representativeness of the benchmark. The dataset contains only 28 base skills, and task selection explicitly favors workflows that admit plausible malicious extensions. This is appropriate for a targeted stress test, but may produce attack rates that do not reflect typical skill usage. Evaluation on independently sampled skills and previously unseen workflows would strengthen the external validity.
  4. Insufficient documentation of evaluation reliability. The paper reports human agreement with the LLM judge on a reviewed subset, but does not specify the subset size, sampling procedure, or detailed agreement statistics. It also does not report confidence intervals or repeated-run variability. Since multiple tasks share the same compromised skill, uncertainty estimates should account for this dependence.
Questions for authors
  1. How do you define and measure stealthiness? Can human reviewers or image-aware detectors reliably identify the injected instructions?
  2. Can you provide a unified and reproducible attack-construction procedure, including the required human effort, number of revisions, and any target-model feedback used during development?
  3. How many runs were manually validated, how were they selected, and what agreement statistics and uncertainty estimates can you provide?
2 comments

We are sincerely grateful for your thorough review. We have carefully addressed all the points you raised, and we hope the revised manuscript has addressed your concerns. For your convenience, all changes are highlighted in red in the revised manuscript.

W1 & Q1: Attack Stealthiness

Thank you for raising this concern. We agree that low conventional-scanner detection rates alone do not establish stealthiness. We therefore add Attack Stealthiness Evaluation subsection: Kimi-K3 and GPT-6.1-sol independently score each attack package's visual and semantic integration, each on a 0--5 scale. Their mean combined score is 7.62/10, indicating that the reviewers generally judge the injected content to be visually consistent with its carrier and naturally integrated into the associated task context. Besides, to assess detectability, we evaluate a Kimi-K2.6 VLM screen that examines complete skill packages, including their visual resources. It flags 72.5% of compromised packages and 4.5% of clean packages, but incurs substantially greater computational cost: screening takes 81.1 seconds per package on average, approximately 255 times the latency of Cisco Skill Scanner (318.0 ms), and consumes 1,118,741 tokens, averaging approximately 9,900 tokens per package.

Furthermore, Appendix E.5 presents an exploratory runtime-defense experiment on 30 cases using Kimi-K3 with Claude Code, with a guardrail based on a PreToolUse hook and deny rules for selected high-risk operations. Although ASR decreases from 76.7% to 46.7%, malicious submissions remain frequent, while legitimate-task success drops from 100.0% to 73.3%. These results suggest that this guardrail can reduce malicious submissions but also substantially disrupt legitimate task completion.

W2 & Q2: Attack Construction

Thank you for this insightful comment. We should have explained our construction process more clearly. NCVA uses a shared human-in-the-loop protocol across skills, with agents assisting implementation and revision. We retain human involvement for two practical reasons: most current LLMs are safety-aligned and will refuse certain requests to construct malicious payloads, and human involvement helps ensure that malicious instructions fit each skill’s workflow and blend naturally into its visual materials, a level of quality that agent-only construction may not consistently achieve.

We have expanded Appendix C.1 to describe the five-step protocol and the criteria for objective selection and quality review, and to provide a Chrome-permissions example. Appendix C.2 reports the authoring roles and effort: a human expert typically spends about 30 minutes on each malicious skill, and each skill undergoes at least one round of revision. GPT-5.6-sol provides design feedback. During development, we also test candidate attacks with Kimi-K3 and use its refusals, misinterpretations, or execution outcomes to guide one round of refinement. These development trials are separate from the formal benchmark evaluation.

W3: Benchmark Representativeness

Thank you for pointing out this limitation. We agree that the original benchmark was relatively small. For reference, Skill-Inject [1], SkillSafetyBench [2], and OpenSkillRisk [3] contain 202, 155, and 263 executable cases, respectively, whereas our original version contained only 108. We have added 16 base skills and 99 executable cases, expanding the dataset to 44 base skills, 69 malicious variants, and 207 cases. The additions include all ten image-bearing skills in the specified AutoVisualSkill source directory and six adaptations from MMSkills. All results from the expanded NCVA evaluation are reported in the revised manuscript.

Additionally, we would like to clarify that our main goal in designing this benchmark is to examine whether visual instructions embedded in multimodal skills can induce agents to perform attacker-specified malicious operations during otherwise legitimate tasks. As described in Section 3.2 of the original manuscript, a malicious skill publisher can design the functionality and procedures of the published skill, so embedding attacks within these workflows is consistent with our threat model. We acknowledge that the phrase “plausible malicious extensions” in Section 4.1 was imprecise. We intended to describe selecting benign tasks that naturally fit each skill’s functionality and workflow, rather than selecting tasks based on how easily an attack could be introduced. We have corrected this wording. Accordingly, our attack rates should not be interpreted as estimates of risk during typical skill usage across the broader ecosystem.

[1] Schmotz et al. (2026). Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. arXiv:2602.20156v3

[2] Jin et al. (2026). SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces.

[3] Liu et al. (2026). OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills. arXiv:2607.20121v2

W4 & Q3: Evaluation Reliability

Thank you for raising this concern. We acknowledge that the original appendix did not sufficiently document the human verification procedure and its agreement with the LLM judge. We have now added these details to Appendix D.1. For each of the original nine model–harness configurations, we randomly selected two cases from each of the five attack categories, yielding 90 manually reviewed runs. Human reviewers assessed attack success and legitimate-task completion using the execution evidence and task artifacts. The initial human judgments agreed with the LLM judge on 89 of the 90 runs (98.9%), with one run containing a disagreement on one criterion. After carefully re-examining the evidence against the predefined scoring criteria, we confirmed that the LLM judge's original decision was correct and retained it.

We also acknowledge the omission of repeated-run variability in the original submission. Given the time constraints, we focused this supplementary evaluation on Kimi-K3 with Claude Code, conducting three independent runs on the same 207 cases. Each run used a fresh agent session and a reset task environment, with the experimental configuration held fixed. Appendix D.2 now reports the following results:

MetricRun 1Run 2Run 3Mean ± SD
ASR42.5%41.5%39.6%41.2 ± 1.5
TSR97.6%97.0%95.2%96.6 ± 1.3
VIAR44.0%43.5%43.5%43.6 ± 0.3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.