- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether code opacity affects how coding agents handle malicious Agent Skills. It evaluates four models and four agent harnesses using six code-representation conditions over 38 Skills and 3,648 runs. The main result is that malicious-Skill detection drops substantially as the implementation becomes more opaque, while implementation-path access remains similar; however, the aggregate increase in unsafe uptake is not statistically significant.
Critical strengths
The evaluation is large-scale and reasonably well controlled. In particular, the paired design keeps the task, documentation, model, and harness fixed while changing the code representation, and the paper evaluates multiple models and real agent harnesses rather than a single isolated classifier.
Critical weaknesses
-
The paper is built around one fairly simple observation: as malicious code becomes more opaque, agents are less likely to recognize it as malicious. The evaluation is large, but the many stages, tables, and breakdowns mostly repeat this same finding rather than adding substantial new insight. I do not think experimental scale alone makes the contribution sufficiently novel for ICLR.
-
More importantly, the paper does not establish what actually causes the detection drop. Accessing the implementation path does not mean that the agent successfully decoded or inspected the recovered payload. Thus, the current experiment cannot distinguish failure to decode from failure to recognize malicious semantics. The downstream effect is also weak: the aggregate increase in unsafe uptake is not statistically significant.
-
As a reviewer who carefully read the full paper, I found this presentation style (AI-style!!!!!!!!) somewhat frustrating. The large amount of highly polished but repetitive material places a substantial burden on the reviewer without a corresponding increase in scientific insight. I would encourage the authors to be more selective and concise, and to respect the reviewer's time by focusing on the analyses that materially advance the central claim.
Questions for authors
-
Can the authors directly verify whether the agent successfully reconstructs or views the unobfuscated payload at O3/O4? Without this evidence, how can the paper distinguish a decoding failure from a semantic-recognition failure?
-
Given that the aggregate increase in unsafe uptake is not statistically significant, what is the main security conclusion beyond the relatively expected observation that opaque code is harder for agents to recognize as malicious?