- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies implicit-tool attacks in coding agents, where persistent configuration edits cause benign, unmodified IDE components to perform sensitive operations after the agent session. It introduces ImpCraft to discover relevant settings, generate attack scenarios, and validate downstream effects. Across 810 trials involving three commercial products and three foundation models, the authors report an 80.1% attack success rate, compared with 0.2% for direct requests. A consequence-guided reasoning skill prevents successful attacks in 90 GPT-5.5 trials.
Critical strengths
-
The paper addresses an important security risk arising from the interaction between agent-generated configuration changes and downstream execution in development environments.
-
The evaluation uses commercial coding-agent products and unmodified extensions, requiring configuration writes, extension execution, and observed effects to establish attack success.
-
Controlled network receivers and canary files enable concrete runtime verification. The appendices describe the experimental protocol and provide the complete defense instructions.
Critical weaknesses
-
The novelty relative to Cuckoo Attack (Liu et al., 2025) and UntrustIDE (Lin et al., 2024) appears insufficiently established. The former already separates agent-mediated configuration writes from delayed execution, while the latter demonstrates execution through configured script paths. Section 7 seems to understate this overlap. Please clarify which mechanisms are new and establish the technical contribution of automated discovery and validation.
-
Section 3.2 allows adversaries to construct task prompts and repository content, but the evaluation does not clearly separate authorized user requests from attacker-controlled instructions. Appendix E also specifies exact configuration changes. Please provide the benign user task, injection entry point, and attacker-controlled artifacts for each case, and clarify why attackers can supply executable assets but cannot directly modify the target configuration.
-
The sandbox-bypass claims need clearer qualification. Appendix C.1 uses trusted workspaces, enables Allow all for Copilot, and places extension execution outside the terminal sandbox. Please specify the authorization guarantee violated, document effective permissions and process boundaries, and explain how these trust and approval settings constrain the conclusions.
-
ImpCraft's methodological claims lack direct evaluation. The paper asserts high recall, reduced inference overhead, and effective skill-guided generation without reporting scanner precision/recall, generation yield, repair effort, or computational cost. Please evaluate the scanner against independently annotated settings and compare rule-only, model-only, and combined variants. Generation ablations should establish what the skill contributes beyond the underlying model and repository templates.
-
The selection of 27 third-party cases from 235 candidate settings remains insufficiently documented. Please report attempted generations, validation failures, revisions, and exclusions. Because offline pre-validation retains executable configuration channels, the reported ASR is conditional on this selection. Please explain case representativeness and delimit the population to which the results apply.
-
The network/local ASR gap may reflect differences in payload visibility. Network cases require agents to encode credential contents into configuration references, whereas local cases place harmful behavior inside referenced scripts. This confounds capability category with the information available during configuration decisions. Please control for malicious-content visibility, inspection requirements, and task framing before attributing the gap to capability-specific safety weaknesses or model alignment improvements.
-
The defense is evaluated only on attack cases, leaving its utility and false-positive rate unknown. Its conservative refusal policy could also block legitimate toolchain setup. Please evaluate benign configuration tasks and report completion rates, false refusals, token usage, and latency. The claims of zero token overhead, zero behavioral interference, and preserved productivity in Appendix D.5 require empirical support.
-
The stealth and persistence claims exceed the demonstrated evidence. The Flake8 wrapper in Appendix E deletes the canary and exits, without establishing that normal linting remains functional. The protocol verifies one trigger after session termination but does not systematically evaluate repeated activation or IDE restarts. Please assess legitimate functionality, user-visible traces, and persistence across relevant lifecycle events, or qualify these claims.
Questions for authors
-
Section 7 distinguishes this work through configuration-mediated capabilities of benign, unmodified extensions. What is the mechanism-level novelty relative to Cuckoo Attack and UntrustIDE? Beyond the implementation details in Appendices B.2-B.3, what quantitative evidence supports the claimed scanner recall, inference savings, and contribution of the generation skill?
-
Section 3.2 permits attacker-constructed prompts and repository content, while Appendix E.3 specifies the exact settings in a README task. In the evaluated trials, how are the authorized user task and attacker-controlled instructions separated? Given the trusted-workspace and approval assumptions and the Extension Host boundary documented in C.1, which authorization guarantee is violated, and how should the sandbox-bypass claim be scoped?