- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MalSpot, a static framework for detecting and localizing malicious code in LLM agent tools. It uses an LLM to identify security-relevant anchors, constructs traces around their input variables, and selects potentially suspicious traces using static checks and LLM assessments. Multiple LLMs then review the selected evidence alongside the full source code and vote on the tool-level decision and malicious location. The paper reports high detection rates and low false-positive rates across three datasets.
Critical strengths
(i)The paper addresses an emerging security problem in agent tools. And I think it's a thoughtful way to handle behavior that may be either legitimate or malicious. (ii) Anchors and traces give the LLMs specific code to inspect and give a place for reviewer to check rather than just a malicious/benign label. (iii)The evaluation is reasonably thorough: it covers both detection and localization, and MalSpot outperforms several traditional and LLM-based baselines on the reported metrics.
Critical weaknesses
(i) AST matching confirms that a proposed anchor exists, but the paper does not report whether Step I proposes anchors in annotated malicious code or whether Step II retains them. (ii) The paper does not explain how MalSpot handles Base64-encoded payloads or dynamically generated code. (iii)The evaluation does not compare MalSpot with code-behavior-based methods such as MalGuard and Ea4mp, leaving its relative performance unclear. (iv) Removing the static checks changes the majority-vote TPR by only 0.58 percentage points and the FPR by 0.08 points. Given these small differences, paired uncertainty estimates and repeated runs would help establish whether the static checks provide a reliable benefit. (v)The paper does not evaluate whether MalSpot can detect malicious payloads hidden by encoding or code generated in one file and executed in another.
Questions for authors
(i) On MalTool, what proportion of malicious tools have an anchor in an annotated malicious span after Step I, and what proportion still have one after Step II? How does this vary across attack categories? (ii)How are functions discovered across files and framework callbacks in Step I? What happens when dispatch is dynamic? (iii) How do the pipeline's latency and token use compare with the Direct baseline, which sends the full code to an LLM?