acceptodds
Under review as a conference paper at ICLR 2027

Opaque Skills, Cautious Agents? Measuring Code-Opacity Effects in Real Agent Harnesses

Abstract

Agent Skills combine natural-language instructions with executable resources, yet the model that selects a Skill may also decide whether its implementation looks safe. We ask whether changing only code representation alters this judgment during an ordinary task in real coding-agent harnesses. Our six-way design includes a no-code anchor (M0) and a semantics-preserving opacity ladder from readable source (O0) to an encrypted self-decoding envelope (O4). We run 3,648 trials crossing four models with Claude Code, Codex CLI, Gemini CLI, and OpenCode over 38 Skills spanning three languages, single- and multi-file structures, nine malicious-behavior classes, and ten heterogeneous benign controls. Malicious-Skill detection falls from 90/448 (20.1%) at O0 to 19/448 (4.2%) at O4, corresponding to a paired change of −15.8 percentage points (Skill-cluster bootstrap 95% CI [−21.7, −9.8]; Holm-adjusted p < 0.001). Implementation-path access changes little, while detection among runs that target the implementation path falls from 24.6% to 5.3%. This places the failure after path access, but does not establish that agents decoded or viewed the recovered O4 payload. Unsafe uptake rises by 4.9 percentage points, but its confidence interval crosses zero (95% CI [−2.0, 12.3]; p = 0.236), and model effects are heterogeneous. Opacity therefore reliably suppresses observable security recognition, but does not uniformly increase unsafe action.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent skills, coding agents, code opacity, code obfuscation, agent safety, malicious skills, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VvPjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether code opacity affects how coding agents handle malicious Agent Skills. It evaluates four models and four agent harnesses using six code-representation conditions over 38 Skills and 3,648 runs. The main result is that malicious-Skill detection drops substantially as the implementation becomes more opaque, while implementation-path access remains similar; however, the aggregate increase in unsafe uptake is not statistically significant.

Critical strengths

The evaluation is large-scale and reasonably well controlled. In particular, the paired design keeps the task, documentation, model, and harness fixed while changing the code representation, and the paper evaluates multiple models and real agent harnesses rather than a single isolated classifier.

Critical weaknesses
  • The paper is built around one fairly simple observation: as malicious code becomes more opaque, agents are less likely to recognize it as malicious. The evaluation is large, but the many stages, tables, and breakdowns mostly repeat this same finding rather than adding substantial new insight. I do not think experimental scale alone makes the contribution sufficiently novel for ICLR.

  • More importantly, the paper does not establish what actually causes the detection drop. Accessing the implementation path does not mean that the agent successfully decoded or inspected the recovered payload. Thus, the current experiment cannot distinguish failure to decode from failure to recognize malicious semantics. The downstream effect is also weak: the aggregate increase in unsafe uptake is not statistically significant.

  • As a reviewer who carefully read the full paper, I found this presentation style (AI-style!!!!!!!!) somewhat frustrating. The large amount of highly polished but repetitive material places a substantial burden on the reviewer without a corresponding increase in scientific insight. I would encourage the authors to be more selective and concise, and to respect the reviewer's time by focusing on the analyses that materially advance the central claim.

Questions for authors
  • Can the authors directly verify whether the agent successfully reconstructs or views the unobfuscated payload at O3/O4? Without this evidence, how can the paper distinguish a decoding failure from a semantic-recognition failure?

  • Given that the aggregate increase in unsafe uptake is not statistically significant, what is the main security conclusion beyond the relatively expected observation that opaque code is harder for agents to recognize as malicious?

Reviewer_GZPwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates whether altering code representation affects an agent's ability to detect malicious intent in real coding-agent harnesses. The evaluation includes 6 opacity designs, 9 malicious behavior classes, and 38 source skills. Results show that code opacity significantly reduces the detection rate of malicious skills, while the uptake rate remains largely unaffected.

Critical strengths

Comprehensive Evaluation: Covers diverse skill categories, 6 opacity designs, 9 malicious behavior classes, multiple programming languages, single/multi-file setups, and varying complexities.

Interesting Findings: Code opacity lowers malicious skill detection, while the uptake rate remains largely unaffected. Agents will execute encrypted code even when they know it is encrypted, exposing a critical vulnerability.

Critical weaknesses

Unclear Motivation: It is unclear why opaque or encrypted skills would exist in practice or why a user would provide them to an agent.

Unrealistic Setting: The evaluation relies on read-only operations rather than dynamic execution and reasoning, which deviates from real-world agent environments.

Ambiguous Scope: Is this testing the agent's decryption ability? If the agent lacks the recovery method (especially with randomized encryption), we can not assume the agent can understand the code.

Missing Details: The writing is overly brief. The paper did not list the 9 malicious behavior classes or provide the complete set of the 38 source skills.

Questions for authors

What is the practical motivation for opaque skills? Why would users or agents encounter encrypted skills in the real world?

What is the source of the 38 skills, and what are the specific functionalities of the benign and malicious skills?

How exactly is skill or code "complexity" defined and measured? What are the criteria for "four small, five medium, and one large"?

Does the agent have access to the decryption methods to recover the original skill code? If not, why should we assume the agent can comprehend the encrypted code, especially when randomized methods are used?

Reviewer_iJgCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on whether coding agents notice malicious code inside Agent Skills. Skills may contain executable scripts, and when the agent uses the skill for implementation, it may trust it and ignore the security risk inside it. The authors build a six-level design. M0 has no script, O0 is readable source, and O1 to O4 apply stronger obfuscation, ending with an XTEA-encrypted wrapper that decodes itself. The program behavior stays the same across levels. They run 3,648 trials crossing 4 models and 4 harnesses: Claude Code, Codex CLI, Gemini CLI, and OpenCode, and 38 Skills (28 malicious and 10 benign). Experimental results show that malicious Skill detection falls from 20.1% at O0 to 4.2% at O4, a paired drop of 15.8 points.

Critical strengths
  • The paper studies an important problem in agentic tasks: while skills help the agent better complete the benign task, they may also bring security risks.
  • It evaluates 4 models * 4 harnesses with 3 languages, single-and multi-file structures, 9 malicious behavior classes, and both malicious and benign skills. The diverse experimental setting better verifies their claims.
  • This paper measures what agents actually do inside real harnesses: whether they open the code, notice the threat, refuse, or try to use it, instead of doing explicit classification. Experimental results show that malicious Skill detection falls from 20.1% at O0 to 4.2% at O4, a paired drop of 15.8 points. Agents still open the implementation path at similar rates. Unsafe uptake rises by 4.9 points, but this change is not significant.
Critical weaknesses
  • The main finding is expected: prior work already shows that obfuscation hurts LLM code understanding and malware detection. And the study stops at benchmarking without investigating defenses
  • Unsafe uptake changes by only 4.9 points, with an interval from −2.0 to 12.3 (p = 0.236). It is already 28.3% at O0 without any obfuscation. So obfuscation clearly changes what agents say, but not clearly whether it will trigger real insecure behaviors such as uptake.
  • Detection coding has unknown recall. Only positive detection events were audited. Negative runs were not checked by independent annotators. Some detections may be missed, and the miss rate could also show the influence between O0 and O4.
  • Each model, harness, Skill, and condition has one run with provider default sampling. The variance from model sampling is not measured. Besides, the skill set is also small.
Questions for authors
  • Most of the drop happens at O1. Detection falls from 20.1% to 11.8% with simple byte arrays, the mildest obfuscation. O3 and O4 are nearly the same (4.0% and 4.2%). It seems that heavy encryption is not needed. Do you have any explanation or hypothesis for that?
  • Why does OpenCode show zero detection in every condition? Is it because it is too weak, or is the experimental setting not suitable for it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.