acceptodds
Under review as a conference paper at ICLR 2027

RuntimeTickets: Source-Grounded Security Obligations for Agent Skills

Abstract

Third-party Agent Skills extend what an autonomous agent can do, but a malicious Skill can hide data exposure and output manipulation inside a legitimate workflow. Reading a Skill's description or granting tool permissions cannot tell whether its execution meets the user's security requirements. We introduce RuntimeTickets, a protocol that enforces and audits these requirements while a Skill runs. We derive a risk model from released malicious-Skill datasets and use it to build adversarial cases with matched benign counterparts. From the task's delivery contract, a language model proposes task-specific security obligations that cite the Skill's source lines they answer, and host approval admits them only as a narrowing of the host's grant. A runtime gate checks the recipient, object and amount of every outbound send before allowing it, and trusted monitors record effects within their observation scope. Tickets link these checks and observations to the approved obligations of each Skill run, and a deterministic decoder then reconstructs what was approved, attempted, observed and verified, keeping violations apart from missing evidence. Enforcement and decoding involve no language model. We evaluate RuntimeTickets at the network egress boundary of two unmodified agent frameworks, measuring harmful effects, unnecessary blocking and task completion, against production Skill scanners and against the same gate holding only the host's grant. Under RuntimeTickets no injected send took effect and no benign run was blocked, while the same gate holding only the host's grant forwarded injected sends that reused the task's recipient, and the scanners either admitted malicious Skills or withheld up to 53.3% of benign ones. Ablations remove the per-task ticket and the effect evidence. RuntimeTickets ties approved requirements to verifiable execution evidence, a common foundation for safer and auditable deployment of third-party Skills.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent security, AI safety, third-party agent skills, auditability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gKMsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RuntimeTickets, a runtime security and auditing framework for third-party LLM agent skills. Instead of relying solely on static skill inspection or deployment-wide permissions, RuntimeTickets generates a task-specific ticket before execution. The ticket specifies allowed recipients, objects, and quantities, is constrained to only narrow the host’s existing authority, and is then frozen before execution. During execution, a model-free gate checks outbound effects against the ticket at the network egress boundary. The system also records authenticated execution evidence and uses a deterministic decoder to classify obligations as SUPPORTED, VIOLATED, or UNKNOWN. The paper evaluates RuntimeTickets using malicious/benign skill twin pairs derived from released datasets, comparing against two skill scanners, a deployment-wide allow-list, and a no-gate setting on OpenClaw and Hermes Agent.

Critical strengths
  1. The paper studies an important and practical problem: static inspection of third-party skills is insufficient to guarantee safe runtime behavior.
  2. The separation between model-based ticket proposal and model-free enforcement is a strong design choice. Once the ticket is frozen, malicious skill instructions cannot directly manipulate the runtime policy.
  3. The narrow-only compilation principle is intuitive and provides a useful safety property: the generated ticket cannot expand the host’s original authority.
  4. The use of malicious/benign twin pairs is a reasonable evaluation methodology for studying both security and overblocking.
Critical weaknesses
  1. The experimental scope is substantially narrower than the general RuntimeTickets formulation. Although the risk model includes exfiltration, destructive changes, access weakening, content manipulation, and safeguard suppression, the actual runtime enforcement experiment mainly evaluates outbound network sends. Local file deletion, permission changes, and safeguard suppression are only observed rather than prevented by the proposed gate.
  2. The defense relies heavily on a structured delivery contract and host-provided validator. In the experiments, the task already specifies the authorized recipient, valid object, and quantity. Many realistic agent tasks do not have such explicit machine-checkable contracts, making it unclear how RuntimeTickets generalizes to open-ended tasks.
  3. The method cannot prevent malicious outputs that satisfy the validator. If an attacker generates a harmful or manipulated object that still passes the host’s format/content validator, uses the authorized recipient, and remains within the permitted quantity, the ticket will allow it. This substantially limits protection against semantic manipulation.
  4. The runtime baselines are relatively weak. RuntimeTickets is mainly compared against no gate and a deployment-wide allow-list. Existing runtime systems such as SkillGuard, AIRGuard, OATS, ToolSafe, or AgentBound are discussed but not experimentally compared, making it difficult to determine the practical advantage over stronger runtime defenses.
Questions for authors

Refer to weaknesses.

Reviewer_DC9gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a novel defense against malicious agent skills. The core idea is to form per-task, model-free enforcement at the egress boundary, with a frozen ticket that narrows the host grant. Experiments and evaluation across a few agent skills demonstrate effectiveness.

Critical strengths
  • interesting approach and well motivated defense

  • The comparison to source scanners, allow-lists, and call-level monitors is great

  • experiments are well designed. Twin pairs are built from released corpora without writing new skills or injections. Calibration and pre-registration are reported. The endpoint audit and exclusion rules are transparent.

Critical weaknesses
  • the variety of scenarios and skills is limited, The main runtime experiment has 32 attack runs and 29 benign runs under RuntimeTickets.

  • The paper compares to two source scanners and an allow-list. It does not compare to runtime monitors like SkillGuard, AIRGuard, ToolSafe, AgentBound, or other related enforcement.

  • The threat model explicitly excludes an attacker who knows about tickets and adapts.

Questions for authors

can you get confidence interval for ASR under RuntimeTickets ?

Can you add at least one adaptive attack?

What does the model add to ticket generation beyond a deterministic template over the delivery contract? Could you ablate the model and use a fixed template?

Reviewer_Kd3iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RuntimeTickets, a runtime security framework for third-party Agent Skills. It uses an LLM to generate task-specific security obligations, which are then checked and enforced by a model-free runtime gate. The framework also maintains authenticated execution records for auditing security violations. Experiments on two agent frameworks show that RuntimeTickets can prevent unauthorized outbound sends while avoiding unnecessary blocking of benign Skills.

Critical strengths
  1. Interesting motivation. The paper addresses an important security problem in Agent Skills and highlights the limitations of existing static scanners and permission-based defenses.
  2. Good writing. The paper is well-written and logically organized. The motivation and overall framework are clearly explained.
  3. Detailed security analysis. The paper provides a clear threat model, multiple attack categories, and detailed analyses of security enforcement and auditability.
Critical weaknesses

Major Weaknesses:

  1. The current framework mainly focuses on controlling outbound network requests. However, malicious Agent Skills can also perform local file modifications, permission changes, or other harmful operations. It remains unclear how well RuntimeTickets can protect against broader attacks.
  2. The paper uses an LLM to generate task-specific security obligations. However, Section 3.3 and Appendix C.1 acknowledge that a deterministic template could generate the same three obligations in the current experiments. This raises concerns about the actual contribution of the LLM-based generation mechanism.
  3. Although the paper reports zero overkill, only 37.9% of benign runs successfully delivered the required output under RuntimeTickets. This raises concerns about practical usability. The current overkill metric may not fully capture how security enforcement affects overall task completion.
  4. Security guarantees rely on strong assumptions about the execution environment. RuntimeTickets assumes that every outbound connection passes through a trusted egress proxy and that the host provides reliable validators. However, these assumptions may not always hold in real-world agent deployments.

Minor Weaknesses:

  1. The paper could provide more examples of generated security tickets to help readers understand how they vary across tasks.
  2. The paper lacks a detailed analysis of runtime overhead, including latency, memory consumption, and ticket-generation costs.
Questions for authors
  1. Why is an LLM needed to generate security obligations when a deterministic template can produce the same rules in the current experiments? What additional benefits does the LLM provide?
  2. How would RuntimeTickets handle malicious operations that do not involve outbound network requests, such as local file deletion or permission changes?
  3. How does the framework perform against adaptive attackers who know the ticket-generation and enforcement mechanisms?
Reviewer_PUFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RuntimeTickets, a protocol for enforcing and auditing task-specific security obligations during third-party Agent Skill execution. An LLM proposes obligations linked to Skill source lines; a trusted compiler and probe approve a ticket bounded by the host’s grant. A deterministic egress gate subsequently checks recipients, object validity, and cumulative quantities. Authenticated records support deterministic reconstruction of decisions and obligation-level verdicts. The main contribution is the integration of task-specific restrictions, network-boundary enforcement, and structured execution auditing.

Critical strengths
  • Combining recipient restrictions with object checks and cumulative limits addresses a meaningful weakness of recipient-only allow-lists. Keeping runtime decisions deterministic also makes enforcement easier to inspect and reproduce.
  • The paper separates gate decisions from observed effects and reports limitations of its live audit evaluation. This is a valuable foundation for more rigorous agent-security reporting.
Critical weaknesses
  • Section 5.1 and Appendix C.2 state that legitimate deliveries use plaintext, while injected sends to granted recipients use TLS. The gate therefore reads legitimate objects but cannot inspect the corresponding malicious objects.
  • Section 3.3 and Appendix C.1 explicitly acknowledge that the structured delivery contract already determines the three standard obligations and that a deterministic template would pass the same probe. The model’s additional contribution is the association of obligations with Skill source lines. The evaluation does not establish the correctness or usefulness of those associations, or quantify their benefit for an auditor. Comparing against a grant-only allow-list therefore establishes the value of adding task restrictions, without isolating the value of source grounding.
Questions for authors
  • Could you provide a sanitized specification of the object validators?
  • How does RuntimeTickets perform when legitimate and malicious sends use the same transport and inspection conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.