- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies an inspection–execution gap in agent skill security. Specifically, a skill can contain benign Python source alongside a manipulated .pyc cache. The scanner approves the visible source, but Python loads the cache and executes different behavior. The paper evaluates whether existing skill scanners catch this mismatch and proposes execution-aware validation (EAV) to check runtime-selected artifacts against trusted compilations of the inspected source. The reported results show that EAV can detect all evaluated source-present cache substitutions and recall can reach 92.8% at a 10.0% false-positive rate across five attack families.
Critical strengths
-
This paper demonstrates a concrete security weakness in agent skills. The paper shows how scanners can approve benign source while the runtime executes a different, concealed bytecode artifact.
-
EAV is a practical defense with controlled evaluation. The paper demonstrates EAV's effectiveness while examining false positives and compatibility limitations.
Critical weaknesses
-
Limited novelty: The authors already acknowledge that Python cache poisoning is already known. The paper needs to establish what new insights arise from applying it to agent skill scanners.
-
Limited evaluation generality: The attack corpus contains 100 curated skills, only 13 of which support placement on an existing script path. The end-to-end runtime evaluation is limited to a single agent harness and a single underlying model.
-
Unclear incremental value of EAV: The results in Table 3 show that trusted reproduction alone detects all 100 tested source-present cache substitutions. A key missing comparison is an existing scanner augmented with the same cache validation.
-
The ablation study provides limited evidence of individual component contributions: the results show that removing several emphasized components barely changes recall.
-
Unclear separation between EAV development and evaluation. The paper does not clearly describe the separation between development and evaluation data. It is unclear whether the reported evaluation datasets were excluded from their development and tuning.
Questions for authors
- What new insight does the agent skill setting reveal beyond known Python cache poisoning?
- How does EAV compare with an existing strong scanner augmented with the same trusted recompilation check?
- Does the attack remain effective across different agents and models while preserving normal task performance?
- Which datasets were used to develop and tune EAV's prompts, rules, weights, and threshold? Were these settings frozen before testing on held-out evaluation data?
Thank you for the additional experiments. They largely address my concerns. My remaining concern is whether EAV adds enough beyond an existing scanner with trusted recompilation. The lower FPR is helpful, although 28% on the new test set remains high for practical use. Given the limited novelty, I am still not convinced that the contribution is sufficient for acceptance, so I will maintain my rating.