- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
LLM agents typically have direct access to shell, file system, browser, and credentials, so it is very important to ensure agent safety. This paper targets agent runtime, and proposes LARA, which aims to synthesize security rules from historical vulnerability fixes to audit local LLM agent runtimes. More specifically, LARA iteratively revises these rules using feedback from CodeQL analysis on vulnerable and fixed revisions, and then adapts them across runtimes. Experimental results show improved detection on construction and refinement cases and 25 previously unreported vulnerability candidates supported by manually executed PoCs.
Critical strengths
The paper targets the timely and important problem of agent safety. It is also great to see this paper target the agent runtime. The idea of leveraging historical vulnerability fixes to audit local LLM agent runtimes is also interesting and can be promising.
The experimental study has been performed on real-world agent systems, including OpenClaw, Hermes, and Nanobot. Moreover, 25 previously unreported vulnerability candidates have been produced with PoC support.
Critical weaknesses
Overall, I feel the paper presentation shall be further improved in the following ways:
-
It is not quite clear to me why the term “fuzzing” is used while the paper is mostly just about “mutation”. “Fuzzing” and “semantic fuzzing” have different meanings in the software literature, and the current presentation can be a bit confusing.
-
The related work section is a bit too brief, and shalle be further augmented.
-
The approach is a bit hard to follow, and some concrete code examples would definitely help there.
-
The experimental setup is also not well justified and explained. For example, it is not quite clear what cases are used for learning and evaluation.
-
How much human effort was required to identify the 25 PoC-supported candidates, and how does this compare with auditing with existing tools or LLMs without LARA? How many of the candidates have been confirmed or commented by the agent developers?
Questions for authors
Pls see the questions in the weaknesses part.
2 comments
Complete Response to Reviewer Comments
Thank you for the comments. We would like to clarify the role of semantic fuzzing and share some additional results addressing the reviewer's concerns.
Semantic Fuzzing and the Semantic-Rule Layer
LARA fuzzes security rules rather than program inputs. It adds, modifies, or removes semantic rules based on vulnerabilities missed in previous rounds, translates the resulting rules into CodeQL queries, and uses validation feedback to guide further refinement.
We work at the semantic level because the same security problem can appear in different code implementations. Rather than directly modifying a query to match a particular code pattern, LARA first modifies the underlying security requirement that the query checks.
To examine whether this semantic layer is useful, we introduced SharedQL, a baseline that directly adds, modifies, or removes CodeQL queries in a shared library using feedback from historical vulnerabilities.
Both approaches can use previous results to improve their rules or queries. The main difference is that SharedQL works directly with CodeQL queries, while LARA first works with semantic security rules.
Direct query modification can achieve better coverage of previously seen cases, but this does not necessarily lead to better detection of unseen vulnerabilities with different implementations.
We evaluated both approaches on 275 development vulnerabilities and 117 held-out test vulnerabilities. The held-out cases were not used for rule refinement, and the final rule and query corpora were frozen before testing.
| Metric | LARA | SharedQL |
|---|---|---|
| DEV Patch Recall | 81.5% | 90.9% |
| Held-out TEST Recall | 47.0% (55/117) | 40.2% (47/117) |
Human Effort per Validated Finding
We agree that human effort should be reported and will add an "Auditing Cost" paragraph.
The approximately 93,000 Hermes findings were first deduplicated into root causes, rather than reviewed individually. We manually screened 270 sampled root causes from Hermes and 1,035 across four projects, identifying 66 potential candidates (6.38%; 90% Wilson CI: 5.24–7.74%). Of these, 32 entered confirmation review and 31 had executed PoCs, resulting in 25 validated findings and six falsified candidates. We also reviewed an additional set of 501 root causes, which will be reported separately.
Agent assistance reduced the manual effort required for screening and PoC development. Many findings could be excluded immediately because they were located in test files or did not involve security-relevant behavior. We estimate approximately 4 hours for initial screening, 7 hours for detailed review, and 3 hours for PoC authoring and validation, or 14 person-hours for the sampled screening and confirmation process.
We will clarify that the 6.38% screening yield is not alert precision, the 25 validated findings do not represent a per-project rate, and we do not claim overall auditing-efficiency superiority.
Comparison with Existing Tools and LLM-Only Auditing
We agree that the paper should distinguish vulnerability localization from end-to-end auditing. Section 7 already describes LARA as a producer of review locations, and we will make this scope clearer in the abstract and conclusion.
On the 275 development repairs, stock Semgrep and CodeQL rules achieved only 1.8% and 0.4% Success, respectively (Table 4), indicating limited coverage of the runtime-specific failure classes studied in this paper.
LARA already uses an LLM during triage. We therefore agree that a rule-guided versus unguided LLM review comparison would be more informative than treating LARA and LLM auditing as entirely separate approaches. We will clarify the scope of our existing comparisons without claiming an end-to-end advantage that has not been measured.
Developer Confirmation and Finding Provenance
We will extend Tables 7–8 with the rule ID responsible for each finding, its origin (random sample or targeted review), and its disclosure status.
As of November 18, 2026, all 25 findings have been reported, and two have been acknowledged by the respective developers. Findings 15–25 on OpenHands were produced from the unchanged Nanobot rule corpus, and we will explicitly identify the corresponding rules to document their provenance.
Response to the Concern About Semantic Fuzzing and Evaluation
Thank you for the comments. We would like to clarify the role of semantic fuzzing and share some additional results.
LARA fuzzes security rules rather than program inputs. It adds or modifies rules based on vulnerabilities missed in previous rounds, translates them into CodeQL queries, and uses the validation results to guide further changes. We work at the semantic level because the same security problem can appear in different code implementations. Instead of directly changing a query to match a particular code pattern, LARA first modifies the security requirement that the query checks.
To examine whether this semantic layer is useful, we added SharedQL, a baseline that directly adds, modifies, or removes CodeQL queries in a shared library using feedback from historical vulnerabilities. Both approaches can use previous results to improve their rules or queries. The main difference is that SharedQL works directly with CodeQL queries, while LARA first works with semantic security rules. Directly modifying queries may make it easier to cover the specific vulnerabilities seen during development, but this does not necessarily lead to better detection of unseen vulnerabilities with different implementations.
We evaluated both approaches using 275 development vulnerabilities and 117 held-out test vulnerabilities. The held-out cases were not used for rule refinement. The results are as follows: