- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper argues that prior skill-injection evaluations implicitly assume that malicious skills are always retrieved and executed. It introduces CORSA, a router-aware optimization method that jointly targets skill retrieval and payload execution in multi-skill environments. CORSA improves retrieval and end-to-end attack success over SKILLJECT and shows some generalization, transferability, and defense-related results.
Critical strengths
-
The paper identifies an important and underexplored assumption in prior skill-injection work.
-
It evaluates attacks in a more realistic multi-skill routing setting.
-
CORSA outperforms the SKILLJECT baseline on retrieval and end-to-end attack success.
-
The experiment includes useful analyses of task generalization, model/scaffold transfer, and defenses.
-
The central insight is relevant for future benchmark design and security evaluation.
Critical weaknesses
-
The comparison relies almost entirely on a single attack baseline, making it difficult to support claims about prior work more broadly. What if SkillJect is a relatively weak method?
-
Main results are primarily shown for GPT-5.4; other models appear mainly in transfer experiments rather than in the main evaluation.
-
The paper does not sufficiently disentangle retrieval success, post-retrieval payload execution, and end-to-end ASR. It is unclear how CORSA improves retrieval at the mechanism level, since the paper provides limited analysis of the resulting skill modifications. It makes it difficult to understand whether CORSA's advantages come from (1) router awareness, (2) stronger skill rewriting by GEPA, or (3) a more sufficient optimization budget.
-
Cross-router transfer lacks a baseline, making the reported ASR and utility difficult to interpret. What are the results from the original, un-attacked models? This is especially important to know for utility. An attack should ideally not affect model utility to preserve its stealthiness. The impact of the attack on normal task performance is also unclear.
-
Overall, the experimental evidence is narrower than the paper’s broader claim that most prior attacks fail because their skills are not retrieved. At least in the main text, there lacks evidence to present this observation.
Questions for authors
-
What is the conditional payload execution rate given that the poisoned skill has been retrieved? How does that compared to other baselines?
-
Is CORSA’s ASR improvement mainly due to higher retrieval probability or stronger payload execution after retrieval?
-
Which specific changes to the skill text improve retrieval, and are they distinct from generic skill rewriting?
-
How does CORSA compare with other existing skill-injection methods under the same routing setup?
-
Please add baseline and no-attack utility comparisons in the cross-router transfer experiments.
-
Would the main conclusions hold across multiple victim models and with matched optimization budgets?
3 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the new router-aware SkillJect and adapted ToolHijacker baselines, the decomposition of retrieval and conditional execution success, and the additional cross-model and cross-router evaluations.
I find the new results substantially more convincing. In particular, the router-aware SkillJect comparison helps establish that CORSA's improvement is not solely attributable to incorporating router feedback, and the conditional execution analysis clarifies that the main gains arise from improved retrieval.
However, two concerns remain:
-
Optimization budget: Although CORSA uses fewer attacker-LLM calls, the authors acknowledge using more router and victim queries. Without a comparable accounting of optimization resources, it remains difficult to determine whether the improvements arise from the proposed methodology or greater optimization effort.
-
User utility: The new cross-router results show noticeably lower user utility for CORSA than SkillJect in several settings (e.g., 17.9% vs. 38.7% on SR-0.6B). This raises concerns about the claim that CORSA preserves benign functionality. The requested no-attack utility baseline is also still missing from these comparisons.
Overall, I acknowledge that the rebuttal substantially strengthens the empirical evidence and addresses several of my original concerns. Nevertheless, I remain somewhat concerned about the fairness of the optimization-budget comparisons and the trade-off between attack effectiveness and legitimate task utility.
Additional No-Attack Utility Results and Matched-Budget Evidence Supporting CORSA
We thank the reviewer for their follow-up and for acknowledging the additional evidence. We address the two remaining concerns below.
-
Optimization budget. To test whether CORSA’s improvements come from its method rather than simply more optimization, we evaluated adapted ToolHijacker under a matched optimization budget. CORSA still achieves 61.5% higher Hit@1 (32.3% vs. 20.0%) and 84.9% higher ASR (22.0% vs. 11.9%). Our ablation also provides further evidence: CORSA’s two-stage optimization achieves 22.0% ASR, compared with 6.2% for joint optimization and 1.8% for task-wise optimization. These results support the benefit of CORSA’s design beyond simply adding router feedback.
-
User utility. We have added the requested no-attack utility baseline to the cross-router comparison, allowing direct comparison with benign operation. When optimized on SR-0.6B: | Evaluation Router | Method | Hit@1 (%) | ASR (%) | Utility (%) | No-Attack Utility (%) | |---|---|---:|---:|---:|---:| | SR-0.6B | CORSA | 38.3 | 23.3 | 17.9 | 40.3 | | SR-0.6B | SkillJect | 10.7 | 8.0 | 38.7 | 40.3 | | BM25 | CORSA | 37.8 | 26.4 | 21.8 | 42.7 | | BM25 | SkillJect | 14.7 | 12.0 | 40.0 | 42.7 | | R3-0.6B | CORSA | 33.3 | 15.9 | 30.2 | 41.3 | | R3-0.6B | SkillJect | 12.0 | 9.3 | 38.7 | 41.3 |
While utility decreases in some cross-router settings, CORSA maintains competitive utility across our main experiments, including cross-model and scaffold transfer and the hard distractor setting.
We addressed the concerns with new experiments and stronger baselines. CORSA more than doubles SkillJect’s ASR (22% vs. 10%) and achieves 60.6% higher ASR even after adding router-aware optimization to SkillJect. It also outperforms adapted ToolHijacker. Across routers, CORSA achieves higher Hit@1 in all 15 cases and higher ASR in 14. New experiments with GLM-5.3 and Opus-4.8 support its effectiveness across victim models, confirming the benefits of jointly optimizing retrieval and execution.
We thank the reviewer for their valuable questions, constructive feedback, and helpful suggestions.
Regarding Q1
The conditional payload execution rate is calculated as ASR/Hit@1. From Table 1, CORSA achieves 68.1% (22.0/32.3), compared with 90.9% (10.0/11.0) for SkillJect.
SkillJect has a higher execution rate after retrieval because it focuses specifically on making the agent execute the malicious payload. CORSA faces the additional challenge of optimizing both retrieval and execution while preserving the skill's legitimate functionality.
Importantly, conditional execution success matters only when the poisoned skill is retrieved. SkillJect reaches the top retrieval position in only 11.0% of tasks, compared with 32.3% for CORSA. As a result, CORSA more than doubles the end-to-end ASR (22.0% vs. 10.0%), which is the main security-relevant metric in our setting.
Regarding Q2
CORSA's improvement comes mainly from better retrieval. Compared with SkillJect, CORSA increases Hit@1 from 11.0% to 32.3%, while its conditional execution rate is lower (68.1% vs. 90.9%). Nevertheless, CORSA more than doubles end-to-end ASR (22.0% vs. 10.0%).
This supports our central argument: optimizing payload execution alone is insufficient when the poisoned skill is rarely retrieved. CORSA addresses this missing step and substantially improves attack success in realistic routing settings.
Regarding Q3
CORSA uses direct router feedback to improve the poisoned skill's ranking among 2,000 competing skills, rather than relying on generic text rewriting.
In Stage A (Section 3.2), GEPA revises the skill's name, description, and instruction body to improve retrieval across tasks in its cluster. After each revision, CORSA evaluates the candidate against the router and uses the retrieval scores to guide further changes.
For example, GEPA may change a broad description like “Assists with coding” to one covering debugging, code editing, and testing. CORSA then checks whether this improves retrieval across the task cluster, keeping changes based on router feedback.
Unlike generic rewriting, CORSA selects changes based on measured retrieval performance. The malicious payload remains unchanged throughout optimization.
Regarding Q4
Most existing skill-injection methods, including SkillJect, focus on inducing malicious payload execution after a skill is selected, without optimizing its retrieval.
Under the same routing setup, CORSA outperforms SkillJect, achieving 32.3% vs. 11.0% Hit@1 and 22.0% vs. 10.0% end-to-end ASR (Table 1).
To strengthen this comparison, we conducted two additional experiments: (i) adapting ToolHijacker's retrieval optimization to skill packages and (ii) building on SkillJect by adding router-aware optimization to its existing optimization loop. Both were evaluated using the same router, 2,000 competing benign skills, seed skills, and victim model (GPT-5.4). We report task-weighted averages across eight task clusters and eight payload categories.
CORSA achieves the highest retrieval and end-to-end attack success across all evaluated methods, demonstrating the advantage of its two-stage optimization over both baselines.
Regarding Q5
Following the reviewer's suggestion, we conducted an additional cross-router evaluation with SkillJect as a baseline, using the same evaluation routers and victim model (GPT-5.4). We report Hit@1, end-to-end ASR, and user utility, which measures successful completion of the original benign task, independently of whether the malicious payload executes. Across all 15 optimization–evaluation router combinations, CORSA achieves higher Hit@1 in every case and higher ASR in 14 of 15 cases.
When optimized on SR-0.6B:
These results shows that CORSA's advantages in retrieval and end-to-end attack success extend across different router architectures, rather than being limited to the router used during optimization.
We will include the full comparison in the revised Table 4.
Regarding Q6 CORSA's advantage extends across victim models (Table 2). New experiments with GLM-5.3 and Opus-4.8 as victim models further support CORSA's advantage over SkillJect: higher ASR on GLM-5.3 (38% vs. 29%) and higher Hit@1 on Opus-4.8 (27% vs. 7%). CORSA also uses fewer attacker-LLM calls (up to 240 vs. 600), though more router and victim queries.