- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RuntimeTickets, a runtime security and auditing framework for third-party LLM agent skills. Instead of relying solely on static skill inspection or deployment-wide permissions, RuntimeTickets generates a task-specific ticket before execution. The ticket specifies allowed recipients, objects, and quantities, is constrained to only narrow the host’s existing authority, and is then frozen before execution. During execution, a model-free gate checks outbound effects against the ticket at the network egress boundary. The system also records authenticated execution evidence and uses a deterministic decoder to classify obligations as SUPPORTED, VIOLATED, or UNKNOWN. The paper evaluates RuntimeTickets using malicious/benign skill twin pairs derived from released datasets, comparing against two skill scanners, a deployment-wide allow-list, and a no-gate setting on OpenClaw and Hermes Agent.
Critical strengths
- The paper studies an important and practical problem: static inspection of third-party skills is insufficient to guarantee safe runtime behavior.
- The separation between model-based ticket proposal and model-free enforcement is a strong design choice. Once the ticket is frozen, malicious skill instructions cannot directly manipulate the runtime policy.
- The narrow-only compilation principle is intuitive and provides a useful safety property: the generated ticket cannot expand the host’s original authority.
- The use of malicious/benign twin pairs is a reasonable evaluation methodology for studying both security and overblocking.
Critical weaknesses
- The experimental scope is substantially narrower than the general RuntimeTickets formulation. Although the risk model includes exfiltration, destructive changes, access weakening, content manipulation, and safeguard suppression, the actual runtime enforcement experiment mainly evaluates outbound network sends. Local file deletion, permission changes, and safeguard suppression are only observed rather than prevented by the proposed gate.
- The defense relies heavily on a structured delivery contract and host-provided validator. In the experiments, the task already specifies the authorized recipient, valid object, and quantity. Many realistic agent tasks do not have such explicit machine-checkable contracts, making it unclear how RuntimeTickets generalizes to open-ended tasks.
- The method cannot prevent malicious outputs that satisfy the validator. If an attacker generates a harmful or manipulated object that still passes the host’s format/content validator, uses the authorized recipient, and remains within the permitted quantity, the ticket will allow it. This substantially limits protection against semantic manipulation.
- The runtime baselines are relatively weak. RuntimeTickets is mainly compared against no gate and a deployment-wide allow-list. Existing runtime systems such as SkillGuard, AIRGuard, OATS, ToolSafe, or AgentBound are discussed but not experimentally compared, making it difficult to determine the practical advantage over stronger runtime defenses.
Questions for authors
Refer to weaknesses.