acceptodds
Under review as a conference paper at ICLR 2027

PyCache Trap: The Inspection–Execution Gap in Agent Skill Scanners

Abstract

Agent skills combine instructions with executable resources, giving third-party packages access to an agent's runtime. Existing skill scanners inspect documentation and visible source, but Python may execute a bundled bytecode cache with different behavior. We study this gap between inspection and execution through PyCache Trap, which pairs benign source with a substituted cache accepted by the loader and connects it to a task-relevant invocation. Scanner-guided rewriting changes the invocation wording while preserving the cache body, separating package admission from recognition of the concealed behavior. Across 100 skills and seven scanners, PyCache Trap achieves 94–100% attack success, with no semantic recognition of the cache-resident behavior. We propose execution-aware validation (EAV) to connect inspected instructions, scripts, imports, and runtime artifacts in a typed execution graph. EAV combines grounded behavioral analysis with trusted reproduction of compiled artifacts. It detects all 100 evaluated source-present cache substitutions and reaches 92.8% Recall at 10.0% FPR across five attack families and 200 benign skills. The results support checking the executable artifacts a runtime can select as part of skill admission, within the supported loaders and code-object normalization. The code is released at https://anonymous.4open.science/r/pycache-trap-artifact-4FA3https://anonymous.4open.science/r/pycache-trap-artifact-4FA3.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent skill security, Software supply-chain security, Execution-aware validation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N7tAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an inspection–execution gap in agent skill security. Specifically, a skill can contain benign Python source alongside a manipulated .pyc cache. The scanner approves the visible source, but Python loads the cache and executes different behavior. The paper evaluates whether existing skill scanners catch this mismatch and proposes execution-aware validation (EAV) to check runtime-selected artifacts against trusted compilations of the inspected source. The reported results show that EAV can detect all evaluated source-present cache substitutions and recall can reach 92.8% at a 10.0% false-positive rate across five attack families.

Critical strengths
  1. This paper demonstrates a concrete security weakness in agent skills. The paper shows how scanners can approve benign source while the runtime executes a different, concealed bytecode artifact.

  2. EAV is a practical defense with controlled evaluation. The paper demonstrates EAV's effectiveness while examining false positives and compatibility limitations.

Critical weaknesses
  1. Limited novelty: The authors already acknowledge that Python cache poisoning is already known. The paper needs to establish what new insights arise from applying it to agent skill scanners.

  2. Limited evaluation generality: The attack corpus contains 100 curated skills, only 13 of which support placement on an existing script path. The end-to-end runtime evaluation is limited to a single agent harness and a single underlying model.

  3. Unclear incremental value of EAV: The results in Table 3 show that trusted reproduction alone detects all 100 tested source-present cache substitutions. A key missing comparison is an existing scanner augmented with the same cache validation.

  4. The ablation study provides limited evidence of individual component contributions: the results show that removing several emphasized components barely changes recall.

  5. Unclear separation between EAV development and evaluation. The paper does not clearly describe the separation between development and evaluation data. It is unclear whether the reported evaluation datasets were excluded from their development and tuning.

Questions for authors
  1. What new insight does the agent skill setting reveal beyond known Python cache poisoning?
  2. How does EAV compare with an existing strong scanner augmented with the same trusted recompilation check?
  3. Does the attack remain effective across different agents and models while preserving normal task performance?
  4. Which datasets were used to develop and tune EAV's prompts, rules, weights, and threshold? Were these settings frozen before testing on held-out evaluation data?
Reviewer_CtZmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PyCache Trap, an attack that exploits the inspection vs. execution gap in Agent Skill scanners by hiding malicious behavior in Python bytecode caches while presenting benign source code for inspection. The authors evaluate the attack against seven existing skill scanners, achieving 94-100% attack success rates. To address this vulnerability, they propose Execution-Aware Validation (EAV), a defense that combines typed execution graphs, behavioral analysis, and executable provenance verification to identify malicious skill behavior. Their experiments show that EAV detects all evaluated source-present cache substitutions and achieves 92.8% recall at a 10% false positive rate across five attack families.

Critical strengths
  • Clear presentation and technical soundness: The paper is well-written and easy to follow, with a clear explanation of the threat model, attack construction, and proposed defense. I particularly liked how the authors distinguish scanner admission from actual runtime execution, which makes the evaluation more rigorous and the results easier to interpret.
  • Comprehensive experimental evaluation: The authors evaluate PyCache Trap against seven existing skill scanners and compare EAV with multiple baselines across different attack families. I also appreciate the additional experiments on attack transferability, runtime execution, and defense ablations.
Critical weaknesses
  1. Limited novelty of the attack mechanism: My main concern is the novelty of PyCache Trap. The underlying mechanism of hiding malicious Python bytecode behind benign source code has already been studied in prior cache-poisoning research, including From Constrictor to Serpent (Ohm et al., 2026). While applying this technique to Agent Skills is interesting, I am not fully convinced that the proposed attack offers substantial technical novelty beyond adapting an existing supply-chain attack to a new setting.

  2. Realism of the threat model: I have some concerns about how realistic the attack setup is. The attack assumes that bundled .pyc files are preserved during skill installation and subsequently executed by the runtime, but the paper does not establish how common this behavior is in real-world Agent Skill environments. Additionally, only 13 out of 100 skills supported placing the attack on an existing script path, while the remaining skills required an explicitly added entry call. I would have liked stronger evidence that these conditions reflect practical Agent Skill workflows.

  3. Limited generalizability of EAV: Although EAV achieves promising results, its strongest component, trusted reproduction, primarily addresses Python source - bytecode mismatches. Agent Skills can also execute JavaScript, shell scripts, native binaries, and dynamically loaded code. I am not convinced that the paper sufficiently establishes how EAV would generalize to these other execution mechanisms, especially when runtime artifacts cannot be resolved statically.

  4. Missing relevant Agent Skill security literature: I think the related-work discussion could be more comprehensive, particularly regarding existing Agent Skill security works. Several relevant works are missing, including [1], which investigates backdoor attacks through malicious Agent Skills; [2], which studies semantic supply-chain attacks targeting Agent Skill registries; and [3], which proposes a security auditing framework for Agent Skill marketplaces. I would have liked a clearer discussion of how PyCache Trap differs from these existing attacks and how EAV advances beyond prior skill-security defenses.

    References:

    • [1] SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems.
    • [2] Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry.
    • [3] SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration.
Questions for authors
  1. Figure 4 shows that the attack success rate is already very high before substantial adaptation. Could the authors clarify how much scanner-guided rewriting actually contributes beyond cache substitution alone?

  2. Do the modified skills continue to perform their original tasks correctly? Reporting a metric like "benign task completion rates" would help establish how practical and stealthy the attacks are.

  3. Have the authors verified whether popular Agent Skill platforms preserve and execute bundled .pyc files during normal installation and invocation?

  4. How do attack success rates differ between the 13 skills with existing script paths and the remaining 87 skills requiring an added entry call?

  5. How would EAV handle custom import hooks or dynamically generated execution paths where the actual executable artifacts cannot be reliably identified before runtime?

Reviewer_Dpt2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate a new mechanism of attack against agentic skill systems that makes use of a quirk in the Python interpreter that caches compiled bytecode. The insight is that there could be a difference between the Python source code and the underlying bytecode that may already have been cached. If an adversary gets malicious bytecode cached, they could pass benign-looking source code through an agent that, when it goes to shell out to the interpreter, ends up executing the malicious bytecode.

The authors show a high attack success rate without any indication of recognition of the semantic difference between the source code and cached byte code. The authors present "Execution Award Validation" to build a new type of graph structure to determine when and how to recompile intermediate artifacts.

Critical strengths
  • This paper shows a very strong, concrete, and straightforward mechanism for attacking agentic systems in a very objective and precise way.

  • The defense mechanism is cheap and deterministic (albeit straightforward -- just rebuild all the artifacts...)

Critical weaknesses
  • The authors assume a situation in which a package is shipped with a cache, and that cached bytecode is actually chosen by the loader to execute. This may be trivially defeated. The authors provide no indication this is a realistic attack pathway.

  • It is not clear what the value of the EAV is when the attack appears to be defeated just by rebuilding the source into the bytecode cache. The EAV seems to serve a secondary purpose of dealing with malicious scripting material received in prompts or input images. Perhaps that is important as well, but it doesn't seem to be relevant to the key question about cached bytecode.

Questions for authors
  • How realistic is the threat model? Have there been skills published that actually exhibit this behavior?

  • What is the value of the EAV graph when it comes to cached bytecode? Is it serving the secondary purpose of understanding or preventing multimodal prompt injection attacks?

Comments on the paper

1 comment

Thank you for the additional experiments. They largely address my concerns. My remaining concern is whether EAV adds enough beyond an existing scanner with trusted recompilation. The lower FPR is helpful, although 28% on the new test set remains high for practical use. Given the limited novelty, I am still not convinced that the contribution is sufficient for acceptance, so I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.