acceptodds
Under review as a conference paper at ICLR 2027

Surviving the Router: Optimizing Skill Injections for Retrieval and Execution

Abstract

AI agents increasingly rely on modular third-party “skills” that are dynamically selected by skill routers to execute complex tasks. While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations of- ten assume settings where the malicious skill is already selected for execution. We show that this assumption can substantially overestimate attack success. In realis- tic multi-skill environments, injected skills must first compete for retrieval, reduc- ing the effective attack success rate (ASR) of existing injections by 87–97%. To address this limitation, we introduce CORSA (Cluster Optimization for Router- Aware Skill Attacks), a router-aware attack that optimizes skill injections for both retrieval and execution across clusters of related tasks. We evaluate skill injection attacks under router-managed multi-skill settings by extending the benchmark in- troduced by SkillRouter with eight malicious payload categories. CORSA uses successive optimization stages to first improve retrieval and then optimize end-to- end attack success, while we evaluate user utility and injection naturalism sepa- rately. Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
security, skills, prompt injections, skill injections, skill router, agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TUJrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper argues that prior skill-injection evaluations implicitly assume that malicious skills are always retrieved and executed. It introduces CORSA, a router-aware optimization method that jointly targets skill retrieval and payload execution in multi-skill environments. CORSA improves retrieval and end-to-end attack success over SKILLJECT and shows some generalization, transferability, and defense-related results.

Critical strengths
  • The paper identifies an important and underexplored assumption in prior skill-injection work.

  • It evaluates attacks in a more realistic multi-skill routing setting.

  • CORSA outperforms the SKILLJECT baseline on retrieval and end-to-end attack success.

  • The experiment includes useful analyses of task generalization, model/scaffold transfer, and defenses.

  • The central insight is relevant for future benchmark design and security evaluation.

Critical weaknesses
  • The comparison relies almost entirely on a single attack baseline, making it difficult to support claims about prior work more broadly. What if SkillJect is a relatively weak method?

  • Main results are primarily shown for GPT-5.4; other models appear mainly in transfer experiments rather than in the main evaluation.

  • The paper does not sufficiently disentangle retrieval success, post-retrieval payload execution, and end-to-end ASR. It is unclear how CORSA improves retrieval at the mechanism level, since the paper provides limited analysis of the resulting skill modifications. It makes it difficult to understand whether CORSA's advantages come from (1) router awareness, (2) stronger skill rewriting by GEPA, or (3) a more sufficient optimization budget.

  • Cross-router transfer lacks a baseline, making the reported ASR and utility difficult to interpret. What are the results from the original, un-attacked models? This is especially important to know for utility. An attack should ideally not affect model utility to preserve its stealthiness. The impact of the attack on normal task performance is also unclear.

  • Overall, the experimental evidence is narrower than the paper’s broader claim that most prior attacks fail because their skills are not retrieved. At least in the main text, there lacks evidence to present this observation.

Questions for authors
  1. What is the conditional payload execution rate given that the poisoned skill has been retrieved? How does that compared to other baselines?

  2. Is CORSA’s ASR improvement mainly due to higher retrieval probability or stronger payload execution after retrieval?

  3. Which specific changes to the skill text improve retrieval, and are they distinct from generic skill rewriting?

  4. How does CORSA compare with other existing skill-injection methods under the same routing setup?

  5. Please add baseline and no-attack utility comparisons in the cross-router transfer experiments.

  6. Would the main conclusions hold across multiple victim models and with matched optimization budgets?

3 comments
AuthorsOpenReview ↗

We addressed the concerns with new experiments and stronger baselines. CORSA more than doubles SkillJect’s ASR (22% vs. 10%) and achieves 60.6% higher ASR even after adding router-aware optimization to SkillJect. It also outperforms adapted ToolHijacker. Across routers, CORSA achieves higher Hit@1 in all 15 cases and higher ASR in 14. New experiments with GLM-5.3 and Opus-4.8 support its effectiveness across victim models, confirming the benefits of jointly optimizing retrieval and execution.

We thank the reviewer for their valuable questions, constructive feedback, and helpful suggestions.

Regarding Q1

The conditional payload execution rate is calculated as ASR/Hit@1. From Table 1, CORSA achieves 68.1% (22.0/32.3), compared with 90.9% (10.0/11.0) for SkillJect.

SkillJect has a higher execution rate after retrieval because it focuses specifically on making the agent execute the malicious payload. CORSA faces the additional challenge of optimizing both retrieval and execution while preserving the skill's legitimate functionality.

Importantly, conditional execution success matters only when the poisoned skill is retrieved. SkillJect reaches the top retrieval position in only 11.0% of tasks, compared with 32.3% for CORSA. As a result, CORSA more than doubles the end-to-end ASR (22.0% vs. 10.0%), which is the main security-relevant metric in our setting.

Regarding Q2

CORSA's improvement comes mainly from better retrieval. Compared with SkillJect, CORSA increases Hit@1 from 11.0% to 32.3%, while its conditional execution rate is lower (68.1% vs. 90.9%). Nevertheless, CORSA more than doubles end-to-end ASR (22.0% vs. 10.0%).

This supports our central argument: optimizing payload execution alone is insufficient when the poisoned skill is rarely retrieved. CORSA addresses this missing step and substantially improves attack success in realistic routing settings.

Regarding Q3

CORSA uses direct router feedback to improve the poisoned skill's ranking among 2,000 competing skills, rather than relying on generic text rewriting.

In Stage A (Section 3.2), GEPA revises the skill's name, description, and instruction body to improve retrieval across tasks in its cluster. After each revision, CORSA evaluates the candidate against the router and uses the retrieval scores to guide further changes.

For example, GEPA may change a broad description like “Assists with coding” to one covering debugging, code editing, and testing. CORSA then checks whether this improves retrieval across the task cluster, keeping changes based on router feedback.

Unlike generic rewriting, CORSA selects changes based on measured retrieval performance. The malicious payload remains unchanged throughout optimization.

Regarding Q4

Most existing skill-injection methods, including SkillJect, focus on inducing malicious payload execution after a skill is selected, without optimizing its retrieval.

Under the same routing setup, CORSA outperforms SkillJect, achieving 32.3% vs. 11.0% Hit@1 and 22.0% vs. 10.0% end-to-end ASR (Table 1).

To strengthen this comparison, we conducted two additional experiments: (i) adapting ToolHijacker's retrieval optimization to skill packages and (ii) building on SkillJect by adding router-aware optimization to its existing optimization loop. Both were evaluated using the same router, 2,000 competing benign skills, seed skills, and victim model (GPT-5.4). We report task-weighted averages across eight task clusters and eight payload categories.

MethodHit@1 (%)ASR (%)
SkillJect11.010.0
Adapted ToolHijacker20.011.9
SkillJect + router-aware optimization24.613.7
CORSA (ours)32.322.0

CORSA achieves the highest retrieval and end-to-end attack success across all evaluated methods, demonstrating the advantage of its two-stage optimization over both baselines.

Regarding Q5

Following the reviewer's suggestion, we conducted an additional cross-router evaluation with SkillJect as a baseline, using the same evaluation routers and victim model (GPT-5.4). We report Hit@1, end-to-end ASR, and user utility, which measures successful completion of the original benign task, independently of whether the malicious payload executes. Across all 15 optimization–evaluation router combinations, CORSA achieves higher Hit@1 in every case and higher ASR in 14 of 15 cases.

When optimized on SR-0.6B:

Evaluation RouterMethodHit@1 (%)ASR (%)Utility (%)
SR-0.6BCORSA38.323.317.9
SR-0.6BSkillJect10.78.038.7
BM25CORSA37.826.421.8
BM25SkillJect14.712.040.0
R3-0.6BCORSA33.315.930.2
R3-0.6BSkillJect12.09.338.7

These results shows that CORSA's advantages in retrieval and end-to-end attack success extend across different router architectures, rather than being limited to the router used during optimization.

We will include the full comparison in the revised Table 4.

Regarding Q6 CORSA's advantage extends across victim models (Table 2). New experiments with GLM-5.3 and Opus-4.8 as victim models further support CORSA's advantage over SkillJect: higher ASR on GLM-5.3 (38% vs. 29%) and higher Hit@1 on Opus-4.8 (27% vs. 7%). CORSA also uses fewer attacker-LLM calls (up to 240 vs. 600), though more router and victim queries.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the new router-aware SkillJect and adapted ToolHijacker baselines, the decomposition of retrieval and conditional execution success, and the additional cross-model and cross-router evaluations.

I find the new results substantially more convincing. In particular, the router-aware SkillJect comparison helps establish that CORSA's improvement is not solely attributable to incorporating router feedback, and the conditional execution analysis clarifies that the main gains arise from improved retrieval.

However, two concerns remain:

  1. Optimization budget: Although CORSA uses fewer attacker-LLM calls, the authors acknowledge using more router and victim queries. Without a comparable accounting of optimization resources, it remains difficult to determine whether the improvements arise from the proposed methodology or greater optimization effort.

  2. User utility: The new cross-router results show noticeably lower user utility for CORSA than SkillJect in several settings (e.g., 17.9% vs. 38.7% on SR-0.6B). This raises concerns about the claim that CORSA preserves benign functionality. The requested no-attack utility baseline is also still missing from these comparisons.

Overall, I acknowledge that the rebuttal substantially strengthens the empirical evidence and addresses several of my original concerns. Nevertheless, I remain somewhat concerned about the fairness of the optimization-budget comparisons and the trade-off between attack effectiveness and legitimate task utility.

AuthorsOpenReview ↗

Additional No-Attack Utility Results and Matched-Budget Evidence Supporting CORSA

We thank the reviewer for their follow-up and for acknowledging the additional evidence. We address the two remaining concerns below.

  1. Optimization budget. To test whether CORSA’s improvements come from its method rather than simply more optimization, we evaluated adapted ToolHijacker under a matched optimization budget. CORSA still achieves 61.5% higher Hit@1 (32.3% vs. 20.0%) and 84.9% higher ASR (22.0% vs. 11.9%). Our ablation also provides further evidence: CORSA’s two-stage optimization achieves 22.0% ASR, compared with 6.2% for joint optimization and 1.8% for task-wise optimization. These results support the benefit of CORSA’s design beyond simply adding router feedback.

  2. User utility. We have added the requested no-attack utility baseline to the cross-router comparison, allowing direct comparison with benign operation. When optimized on SR-0.6B: | Evaluation Router | Method | Hit@1 (%) | ASR (%) | Utility (%) | No-Attack Utility (%) | |---|---|---:|---:|---:|---:| | SR-0.6B | CORSA | 38.3 | 23.3 | 17.9 | 40.3 | | SR-0.6B | SkillJect | 10.7 | 8.0 | 38.7 | 40.3 | | BM25 | CORSA | 37.8 | 26.4 | 21.8 | 42.7 | | BM25 | SkillJect | 14.7 | 12.0 | 40.0 | 42.7 | | R3-0.6B | CORSA | 33.3 | 15.9 | 30.2 | 41.3 | | R3-0.6B | SkillJect | 12.0 | 9.3 | 38.7 | 41.3 |

While utility decreases in some cross-router settings, CORSA maintains competitive utility across our main experiments, including cross-model and scaffold transfer and the hard distractor setting.

Reviewer_LHqNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies skill-injection attacks in realistic multi-skill environments, where malicious skills must first compete for retrieval before their payloads can execute. It proposes CORSA, a router-aware attack that uses two-stage optimization across clusters of related tasks to improve both retrieval and execution. Experiments show that CORSA improves retrieval and end-to-end attack success over SkillJect while preserving user utility. The attacks also transfer across different routers and LLM backbones.

Critical strengths
  1. The paper identifies an important limitation in existing skill injection evaluations by considering skill routing in realistic multi-skill environments.
  2. The experiments cover a wide range of attack types, routers, victim models, and agent scaffolds.
Critical weaknesses
  1. The paper does not sufficiently distinguish CORSA from existing router-aware attacks, such as ToolHijacker. The related work discussion is brief, and the experiments lack comparisons with router-aware baselines.
  2. CORSA does not seem to fully build on existing skill-injection methods. Its ASR/Hit@1 ratio is lower than SkillJect's, despite better retrieval performance.
Questions for authors
  1. ToolHijacker (NDSS 2026) also uses a two-stage optimization strategy for tool retrieval and selection. What are the key methodological differences between CORSA and ToolHijacker?
  2. How realistic is the assumption that attackers can access the target router's scoring function? Would CORSA remain effective without such access?
  3. The motivation for considering skill routers is clear. However, what is the rationale behind the design of the router attack module in CORSA?
  4. SkillJect has a higher ASR/Hit@1 ratio than CORSA (90.9% vs. 68.1%). Why does CORSA have a lower execution success rate after retrieval? Why not build on SkillJect's attack strategy and add router-aware optimization?
3 comments
AuthorsOpenReview ↗

We addressed the concerns through new experiments and analysis. Under matched budgets, CORSA outperforms adapted ToolHijacker by 61.5% in retrieval and 84.9% in ASR. Even after adding router-aware optimization to SkillJect, CORSA achieves 60.6% higher ASR. Our ablations show the benefit of two-stage optimization (22.0% vs. 6.2% ASR). We also clarified attacker access and demonstrated cross-router transfer. Together, these results show CORSA’s advantages over existing attack methods.

We thank the reviewer for their valuable questions, constructive feedback, and helpful suggestions.

Regarding Q1

ToolHijacker optimizes tool descriptions for retrieval and selection, without evaluating whether the selected tool performs the advertised task. Its attack succeeds when the malicious tool is selected. In contrast, CORSA must get the poisoned skill retrieved, get the agent to execute the hidden malicious payload, and preserve the skill's legitimate functionality. Thus, CORSA measures actual payload execution rather than just skill selection.

Main differences:

First, attack objectives. ToolHijacker aims to make an agent retrieve and select a malicious tool. CORSA targets skills, where selection alone is not enough. The agent must also follow the malicious instructions and execute the hidden payload. For example, with GPT-5.4, CORSA achieves 32.3% retrieval success (Hit@1), but only 22.0% end-to-end attack success (ASR). This gap shows that successful retrieval does not guarantee payload execution, which motivates CORSA's second optimization stage.

Second, attack surfaces and constraints. ToolHijacker creates a malicious tool and optimizes its description to make the agent select it. Since the tool's implementation is already malicious, its description only needs to convince the agent to choose it. CORSA instead modifies an existing legitimate skill. The skill must still perform its original task while hiding malicious instructions that convince the agent to execute the payload. This is a more challenging and realistic supply-chain attack, where users may install a useful skill without knowing it contains malicious instructions.

Third, generalization across tasks. ToolHijacker optimizes a malicious tool for a specific target task, whereas CORSA optimizes one poisoned skill for a cluster of related tasks. We also evaluate CORSA on unseen task variations without further optimization (Table 1). This reflects a realistic setting where attackers cannot predict the exact requests future users will make.

New experiment. We adapted ToolHijacker's approach to skill packages and compared it with CORSA under a matched optimization budget, using the same 2,000 competing benign skills, router, seed skills, and models. Following the reporting protocol of our main results, we report task-weighted averages across eight task clusters and eight payload categories using GPT-5.4.

MethodHit@1 (%)ASR (%)
Adapted ToolHijacker20.011.9
CORSA (ours)32.322.0

CORSA achieves 61.5% higher retrieval success and 84.9% higher end-to-end ASR than adapted ToolHijacker.

Regarding Q2

CORSA requires only black-box access to router scores, not the router's internal weights or architecture (Section 3.1). This is a realistic assumption because attackers can use publicly available routers, such as BM25, OpenAI's text-embedding-3-large, or open-source embedding models, to optimize poisoned skills.

Importantly, CORSA does not require access to the victim's exact router. Our cross-router experiments (Section 5.3, Table 4) show that attacks optimized on one router also succeed against others, without further optimization. For example, an attack optimized on SR-0.6B achieves 26.4% ASR on BM25 and 15.9% on R3-0.6B.

Thus, even when the victim's router is unknown or inaccessible, an attacker can optimize CORSA using a publicly available router and transfer the resulting poisoned skill to another router. We will clarify this distinction in the revision.

Regarding Q3

CORSA's design follows a simple principle: a poisoned skill must first be retrieved by the router and then convince the agent to execute its malicious payload. We make three design choices to achieve this.

First, two-stage optimization. We optimize retrieval first, followed by end-to-end attack success. This is because a payload cannot execute if the poisoned skill is never retrieved. Our ablation (Appendix C.1, Fig. 4) confirms this choice: optimizing both objectives together achieves only 6.2% ASR, compared with 22.0% for CORSA.

Second, cluster-level optimization. We optimize one poisoned skill across several related tasks rather than creating a separate attack for each task. This reflects a realistic setting where attackers publish a skill once, without knowing exactly what future users will ask. Our ablation (Appendix C.1, Fig. 4) shows that task-wise optimization achieves only 1.8% ASR, compared with 22.0% for CORSA.

Third, GEPA optimization. We use GEPA because it can improve the skill's natural-language instructions using feedback from the router and agent, without requiring model weights or gradients. It also maintains candidates that perform well across different tasks, making it suitable for our cluster-level objective.

AuthorsOpenReview ↗

Showing that CORSA more than doubles SkillJect’s end-to-end ASR (22.0% vs. 10.0%) and achieves 60.6% higher ASR even after building on SkillJect by adding router-aware optimization, demonstrating the advantage of CORSA’s two-stage approach.

Regarding Q4

While SkillJect has a higher ASR/Hit@1 ratio, CORSA achieves more than twice its end-to-end attack success (22.0% vs. 10.0%). SkillJect succeeds more often once retrieved, but reaches the top retrieval position in only 11.0% of tasks, compared with 32.3% for CORSA.

CORSA's ratio is lower since SkillJect focuses on execution and can optimize its instructions specifically to make the agent run the malicious payload. CORSA faces the additional challenge of getting the poisoned skill ranked first among 2,000 competing benign skills, while preserving its legitimate functionality. Unlike SkillJect, CORSA must optimize both retrieval and payload execution, making the attack more challenging.

New experiment. Following the reviewer's suggestion to build on SkillJect and add router-aware optimization, we extended SkillJect by incorporating retrieval feedback into its existing optimization loop, while keeping its execution-focused refinement strategy. We compared this adaptation with CORSA using the same router, 2,000 competing benign skills, seed skills, and victim model (GPT-5.4). Following our main evaluation protocol, we report task-weighted averages across eight task clusters and eight payload categories.

MethodHit@1 (%)ASR (%)
SkillJect11.010.0
SkillJect + router-aware optimization24.613.7
CORSA (ours)32.322.0

Adding router-aware optimization improves SkillJect's performance. However, CORSA still achieves 60.6% higher end-to-end ASR (22.0% vs. 13.7%), demonstrating the advantage of CORSA's two-stage optimization.

Thank you for the detailed responses and additional experiments. I have no further questions. However, I still have concerns about the novelty of the proposed method, so I will keep my score.

Reviewer_EuKWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces CORSA, a router-aware skill injection attack that optimizes malicious skills for both retrieval and payload execution. It uses GEPA with cluster-level, two-stage optimization and evaluates attacks across routers, victim models, and agent scaffolds.

Critical strengths
  1. Practical problem formulation. The paper highlights an important limitation of skill-injection evaluations that assume the malicious skill is already loaded. Explicitly considering competitive retrieval provides a more realistic assessment of end-to-end attack effectiveness in router-managed systems.

  2. Broad empirical evaluation. The experiments cover eight payload categories and assess transfer across victim models, router architectures, and agent scaffolds. Evaluating retrieval success, payload execution, and benign task utility separately helps distinguish where attacks succeed or fail.

  3. Useful ablations and analysis. Comparisons with task-wise and joint optimization support the cluster-level, staged design. The finding that optimizing stealthiness improves retrieval but reduces end-to-end ASR also illustrates a meaningful tension between attracting selection and inducing execution.

Critical weaknesses
  1. Limited methodological novelty. ToolHijacker [1] already studies malicious tool documents targeting retrieval and selection through two-phase optimization. CORSA extends this direction to skill retrieval and subsequent payload execution, but primarily combines an existing optimizer with task clustering and staged objectives. The paper does not sufficiently establish a substantive methodological advance over prior retrieval-aware attacks, and a comparison with an adapted ToolHijacker baseline is missing.

  2. Limited generalization beyond optimization tasks. The headline results use the tasks involved in optimization. ASR drops from 22.0% on these tasks to 11.5% on paraphrases and 5.4% on new synthetic tasks. Although CORSA remains stronger than SkillJect, this decline limits evidence that a published malicious skill can reliably compromise diverse downstream requests.

  3. Insufficient optimization-budget reporting. The paper does not clearly specify stage-wise search budgets, stopping criteria, or total router and victim queries. Without matched-budget comparisons, it is difficult to determine how much improvement comes from the proposed design rather than additional optimization effort.


[1] Shi, Jiawen, et al. “Prompt Injection Attack to Tool Selection in LLM Agents.” NDSS, 2026. arXiv:2504.19793.

Questions for authors
  1. What distinguishes CORSA from adapting ToolHijacker to skill packages, and how would such an adaptation perform under a matched budget?

  2. Can the authors provide complete optimization costs and evaluate generalization on additional held-out tasks beyond LLM-generated variations?

1 comment
AuthorsOpenReview ↗

We addressed the concerns with new experiments and analysis. Under matched budgets, CORSA outperforms adapted ToolHijacker by 61.5% in retrieval and 84.9% in ASR. Our ablations confirm the benefit of two-stage optimization (22.0% vs. 6.2% ASR). On held-out tasks, CORSA transfers without re-optimization, while SkillJect fails to reach the top 20. These results reinforce CORSA’s effectiveness, design, and generalization.

We thank the reviewer for their valuable questions, constructive feedback, and helpful suggestions.

Regarding Q1

ToolHijacker optimizes tool descriptions for retrieval and selection, without evaluating whether the selected tool performs the advertised task. Its attack succeeds when the malicious tool is selected. In contrast, CORSA must get the poisoned skill retrieved, get the agent to execute the hidden malicious payload, and preserve the skill's legitimate functionality. Thus, CORSA measures actual payload execution rather than just skill selection.

Main differences:

First, attack objectives. ToolHijacker aims to make an agent retrieve and select a malicious tool. CORSA targets skills, where selection alone is not enough. The agent must also follow the malicious instructions and execute the hidden payload. For example, with GPT-5.4, CORSA achieves 32.3% retrieval success (Hit@1), but only 22.0% end-to-end attack success (ASR). This gap shows that successful retrieval does not guarantee payload execution, which motivates CORSA's second optimization stage.

Second, attack surfaces and constraints. ToolHijacker creates a malicious tool and optimizes its description to make the agent select it. Since the tool's implementation is already malicious, its description only needs to convince the agent to choose it. CORSA instead modifies an existing legitimate skill. The skill must still perform its original task while hiding malicious instructions that convince the agent to execute the payload. This is a more challenging and realistic supply-chain attack, where users may install a useful skill without knowing it contains malicious instructions.

Third, generalization across tasks. ToolHijacker optimizes a malicious tool for a specific target task, whereas CORSA optimizes one poisoned skill for a cluster of related tasks. We also evaluate CORSA on unseen task variations without further optimization (Table 1). This reflects a realistic setting where attackers cannot predict the exact requests future users will make.

New experiment. Following the reviewer's suggestion, we adapted ToolHijacker's approach to skill packages and compared it with CORSA under a matched optimization budget, using the same 2,000 competing benign skills, router, seed skills, and models. Following the reporting protocol of our main results, we report task-weighted averages across eight task clusters and eight payload categories using GPT-5.4.

MethodHit@1 (%)ASR (%)
Adapted ToolHijacker20.011.9
CORSA (ours)32.322.0

CORSA achieves 61.5% higher retrieval success and 84.9% higher end-to-end ASR than adapted ToolHijacker.

Regarding Q2

Optimization budget. CORSA optimizes one skill per cluster (8 skills across 75 tasks), using two stages of up to 15 rounds each. SkillJect instead optimizes one skill per task (75 skills), with up to 8 rounds per task.

Query costs. CORSA uses fewer attacker-LLM calls than SkillJect (up to 240 vs. 600), but more victim executions (approximately 1,200 vs. 600) and router queries (approximately 2,400 vs. 75), since it evaluates candidates across multiple tasks and optimizes both retrieval and execution.

To assess the benefit of this design, our ablations compare task-wise optimization (1.8% ASR) and single-stage joint optimization (6.2% ASR) against CORSA's two-stage approach (22.0% ASR; Appendix C.1, Fig. 4). These results suggest that CORSA's improvement comes not only from the optimization effort, but also from how that effort is used.

Held-out Tasks. To further evaluate CORSA's ability to generalize to unseen tasks, we conducted additional experiments using held-out tasks from the original benchmark, rather than LLM-generated variations.

We evaluated two settings: (i) a 70/30 train–test split and (ii) 3-fold cross-validation, where each task is evaluated without being used during optimization. CORSA achieves 3–5% ASR in both settings, while SkillJect fails to appear even among the top 20 retrieved skills.

Unlike the paraphrase and synthetic evaluations in Table 1, these experiments hold out original benchmark tasks during optimization, providing a separate test of generalization. Even within the same cluster, unseen tasks may require different skills, making it harder for a poisoned skill to rank above competing benign skills. Despite this challenge, CORSA successfully transfers attacks to entirely unseen tasks without further optimization, demonstrating a clear advantage over SkillJect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.