acceptodds
Under review as a conference paper at ICLR 2027

TRACE-GUARD: Stateful Runtime Enforcement for Multi-Step Tool-Using Agents

Abstract

Multi-step safety risks in tool-using LLM agents are history-dependent: the same action can be safe after one sequence of actions but harmful after another. Existing defenses check actions using history or state. The challenge is to identify when earlier actions make the next action unsafe, before it is executed. We present TRACE-GUARD, a runtime defense that maintains a compact, structured safety state encoding safety-relevant information from earlier guard-approved actions. Before dispatch, it evaluates how a proposed action would change this state and decides whether to allow its execution. To evaluate runtime safety beyond end-task outcomes, we introduce RMSBench (Runtime Multi-Step Safety Benchmark), a 300-case executable benchmark for state-dependent runtime safety, with task-specific environment criteria and transition-level runtime evidence. Experiments on RMSBench across four models show that TRACE-GUARD substantially reduces task-criterion ASR (on DeepSeek-V3.2, mean ASR drops from 97.7% to 20.5%). Across 30 executable controlled-history pairs with identical final actions but different preceding histories, TRACE-GUARD distinguishes safe from dangerous cases before dispatch, demonstrating history-conditioned runtime enforcement beyond action-local checking.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent safety, multi-step attacks, stateful runtime enforcement, tool use, physically grounded evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CGdQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-step safety risks in tool-using LLM agents, where the safety of an action depends on previous execution history. The authors propose TRACE-GUARD, a stateful runtime guard that maintains a compact safety state and evaluates candidate state transitions before tool execution. They further introduce RMSBench, an executable benchmark for evaluating runtime safety with transition-level evidence. Experiments show that TRACE-GUARD improves over action-local defenses and can distinguish safe and unsafe cases with identical final actions but different histories.

Critical strengths
  1. Clear formulation of history-dependent runtime safety. The paper identifies an important limitation of action-local safety checking and formulates multi-step agent safety as a state-dependent runtime enforcement problem. The proposed state-transition view provides an interpretable mechanism for tracking accumulated safety-relevant conditions and making pre-dispatch decisions.
  2. Strong controlled evaluation for analyzing the proposed mechanism. The introduced controlled-history experiments are well designed: they keep the final action unchanged while varying only the preceding trajectory, directly testing whether the guard can capture history-dependent risks. The transition-level evidence collected in RMSBench also provides more diagnostic analysis than only reporting final task success or failure.
Critical weaknesses
  1. Limited evidence for generalization beyond manually specified safety policies. The proposed approach relies on a manually designed abstraction of safety-relevant states and transitions. While this design is effective for capturing predefined multi-step risks, it remains unclear how well the approach scales to open-ended agent environments with diverse and evolving safety requirements. In particular, the current evaluation focuses on a fixed rulebook and benchmark-specific scenarios, where the safety states and transition rules can be explicitly specified. The broader claims in the abstract and introduction appear somewhat general, but the paper does not provide sufficient evidence that the proposed state-transition framework can generalize beyond such controlled policy spaces. It would be valuable to evaluate how much effort is required to extend the rulebook to unseen environments or policies, and whether the approach remains effective under such settings.
  2. Missing comparison with stronger history-aware runtime guards. The experiments convincingly show that maintaining safety state is better than purely action-local checking. However, this comparison does not fully establish the advantage of the proposed structured state representation over other history-aware approaches. For example, a strong baseline that directly conditions on the full execution trajectory (or a learned trajectory summarization mechanism) could potentially capture similar history-dependent risks without requiring manually specified state abstractions. Since the main contribution is not simply that "history matters", but rather that explicit structured state transitions are preferable, a comparison against such trajectory-aware guards would be important to validate the proposed design choice.
Questions for authors
  1. Evaluation on existing multi-step safety benchmarks. Given the close connection between TRACE-GUARD and existing multi-step tool-use safety benchmarks such as STAC, why did the authors not evaluate TRACE-GUARD on these established benchmarks?
  2. Scalability of state abstraction. The current implementation uses a finite set of manually designed safety states and transitions. How do the authors envision constructing and maintaining such abstractions for large-scale open-world agents with many tools, resources, and interacting objects?
1 comment

Thank you for recognizing the formulation of history-dependent runtime safety and the controlled evaluation that fixes the final action while varying prior history. Your comments raise two important questions: what explicit state contributes, and how policy rules can scale.

On explicit state and history-aware guards.

Our starting point is that access to history and consistent enforcement of safety conditions are different properties. Under No Defense, the agent processes task history through its native context mechanism, yet ASR remains high. Making history available for task execution therefore does not necessarily make cross-step safety conditions stable decision constraints.

TRACE-GUARD represents these conditions as explicit policy state and checks candidate transitions before dispatch. Across 30 controlled pairs, the final action and deterministic predicate are fixed while the executed prefix changes. The stateful guard correctly distinguishes all 30 pairs; the stateless guard has no pair with both sides correct. Disabling semantic translation allows this experiment to directly test state-conditioned rule selection.

The explicit representation also makes it possible to inspect which prior condition a decision depends on, which rule matched, and which successor state was considered. For example, an archive precondition before deletion can be expressed as an inspectable cross-step policy. This is a concrete design property demonstrated by the implementation and experiments.

A dedicated LLM guard that assesses the full history is a meaningful comparator. No Defense motivates the problem, while the controlled-history experiment tests the state mechanism; neither substitutes for a direct comparison with that guard.

Q1. How should RMSBench be understood relative to existing multi-step safety benchmarks?

RMSBench is designed to connect tasks with execution environments, task-specific before/after checks, and runtime decision records. This allows us to answer both whether the specified task criterion was satisfied and whether the monitor explicitly intervened.

The execution and recording protocol distinguishes harmful-task non-completion, explicit action blocking, and approved actions that fail to produce the target outcome. The reported results should be interpreted under that protocol and its task criteria, including the corresponding scope of proxy or incomplete checks. This evaluation should not be treated as an independent result under an original external benchmark’s protocol.

Q2. How can manually specified states and rules scale to open environments?

Our subsequent research direction has two connected components.

The first is rule templating. Rules can be organized around operation categories, resource attributes, preconditions, and risk transitions, allowing different tools to share policy semantics. For example, deletion of files, database entries, and cloud objects could share checks on target identity, authorization scope, and protective preconditions while using different tool mappings. Each mapping would still require validation against the environment’s semantics.

The second is a self-evolving rule module. New trajectories could reveal conditions missing from existing templates and suggest candidate rules. Independent validation, benign-task regression checks, and versioned updates would determine which candidates enter the rulebook. This offers a direction for reducing manual maintenance while preserving rule provenance and inspectable decisions.

Both directions benefit from resource-scoped state and execution feedback. Refining “archiving occurred” into “the archive of object A was successfully completed,” for example, would better support interleaved operations on multiple objects. The present work provides an implementation and controlled mechanism evidence under fixed policies; template transfer, self-evolving updates, and maintenance costs are subjects for subsequent research.

Reviewer_9nbBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies history-dependent safety risks in multi-step tool-using agents, where the same action can be safe or harmful depending on prior execution history. It proposes TRACE-GUARD, a runtime defense that summarizes each proposed action into a structured representation of its safety-relevant attributes. The defense tracks a symbolic safety state across steps, checks whether the proposed action would move this state into an unsafe one, and decides whether to block the action before execution. The authors also introduce RMSBench, a 300-case executable benchmark whose harmful cases are adapted from STAC. Experiments show that TRACE-GUARD reduces attack success rate on all four evaluated models (e.g., from 97.7% to 20.5% on DeepSeek-V3.2), but at the cost of a reduced benign completion rate. In 30 controlled pairs with identical final actions but different histories, the stateful guard decides all pairs correctly while the stateless guard decides none correctly.

Critical strengths

Strengths

  1. Clear problem formulation. The paper targets a real gap in the current safeguard: individually benign actions can compose into harmful outcomes, which action-local guards cannot detect by design. Modeling this as state-conditioned transitions is intuitive.
  2. Careful evaluation design. The controlled-history pairs ablation isolate the effect of state by fixing the final action and varying only the history. The evaluation also separates environment outcomes from explicit guard decisions, so agent failures are not counted as defense successes.
Critical weaknesses

Weaknesses

W1. Generalization is not established.

The rulebook was developed with access to the benchmark tasks, which risks test-set leakage. Moreover, evaluation is limited to the authors’ own benchmark, whose harmful cases are all adapted from STAC. It is therefore unclear whether the hand-written rules transfer to unseen attacks. I would expect results on at least one external benchmark with multi-step attacks, such as MT-AgentRisk.

W2. Practical value is not demonstrated.

With TRACE-GUARD, benign completion drops on all models, and severely on Qwen (97.9% → 18.8%) and Claude (99.6% → 15.0%). On DeepSeek, the proposer–executor split alone (without the guard) already reduces it to 72%, possibly because information is lost between steps in multi-step tasks. A guardrail that reduces task completion this much raises serious doubts about its practicality. The paper does not analyze the cause of this drop, does not report the same ablation on other models, and does not state which model plays for each role. The cost on task completion is also masked by BSU: a refuse-everything system would score 50% and higher than No Defense (49.6%), so the metric can be redesigned. In addition, the method involves multiple LLM calls per step, yet no latency or cost is reported, making it hard to estimate its performance in realistic settings. Finally, the paper lacks the most direct baseline, an LLM-based guard given the full history, so the benefit of the explicit state machine remains unclear.

Questions for authors

Q1. Which model is used for each role (proposer, semantic translator, executor)? In the cross-model experiments, are all roles replaced with the same model?

Q2. What is the benign completion rate of Proposer Only on Qwen and Claude? Can the authors identify where most benign failures come from, e.g., proposer errors, executor refusals, or information loss between steps?

Q3. How many LLM calls, tokens, and how much latency does TRACE-GUARD require per task on average, compared with the main baselines?

Q4. How does TRACE-GUARD perform on an external benchmark with multi-step attacks, such as MT-AgentRisk, using the frozen rulebook?

Q5. How does TRACE-GUARD compare with an LLM-based guard that is given the full interaction history?

Q6. Would the conclusions still hold under a non-compensatory metric, such as the harmonic mean of (1 − ASR) and BCR? At minimum, could the authors report the refuse-everything baseline (BSU = 50%)?

Please also see the Weaknesses section for other concerns.

2 comments

Thank you for the detailed questions. We agree that safety gains should be assessed together with benign completion.

Q1. Role configurations.

Only the evaluated execution agent changes across DeepSeek, GPT, Qwen, and Claude. The proposer, semantic translator, and evaluation-role model configurations remain fixed. The table names the execution-agent model, not a model shared by all roles. This compares execution agents within the same proposal, monitoring, and evaluation framework.

Q2. Benign completion and proposer design.

No Defense achieves BCR of 97.1%–99.6%, whereas Proposer Only in the DeepSeek mechanism study already has BCR of 72.1%. The proposal–execution interface is therefore an important improvement target. In the same study (Table 24), Stateless Guard has ASR/BCR of 36.8%/72.5%, versus 24.2%/69.6% for Stateful TRACE-GUARD: retaining state reduces ASR by 12.6 percentage points while BCR decreases by 2.9 points.

We will continue improving task decomposition and transfer of goals, dependencies, and execution feedback. Corresponding Proposer Only results for Qwen and Claude are unavailable, so the DeepSeek ablation guides this work rather than establishing the cause of their failures.

Q3. Calls, tokens, and latency.

The following is a budget illustration, not a measured average. For NN valid candidate actions, assuming no repair calls, retries, or cache hits, proposal and translation require 2N2N calls. Relative to Proposer Only with the same action sequence, the guard adds NN translator calls. Deterministic extraction and state-rule checks require no LLM calls. Execution-agent and evaluation calls are additional; BLOCK skips the blocked action's execution.

For five actions, these two components require ten calls. Assuming 800 input tokens per proposer call and 400 per translator call gives 6,000 input tokens. Configured output caps of 160 and 200 tokens per call give a combined cap of 1,800 output tokens. The guard-only increment is five calls, approximately 2,000 input tokens, and at most 1,000 output tokens.

Assuming two seconds per call gives ten seconds for translation, or twenty seconds for proposal plus translation, excluding execution, tools, and evaluation. Actual overhead depends on service latency, prompts, retries, and caching. End-to-end overhead versus No Defense also depends on changes to task execution and cannot be inferred from this estimate alone.

Q4. External evaluation and rule generalization.

The reported evaluation uses fixed rules; external-benchmark performance requires separate evidence. Our subsequent research direction is reusable templates separating operations, resource attributes, preconditions, and risk transitions. A self-evolving rule module could propose candidates from new trajectories, validate them independently, and apply regression checks before versioned updates. Freezing each evaluated version would separate rule development from generalization assessment. These are research directions, not additional evaluation results.

Q5. Full-history guards.

No Defense retains the agent's native history mechanism, yet ASR remains high (97.7% for DeepSeek). History availability alone therefore does not ensure consistent safety checks. TRACE-GUARD explicitly evaluates candidate transitions against policy state. With identical final actions and deterministic predicates, it correctly distinguishes all 30 controlled-history pairs.

A dedicated full-history LLM guard performs a separate safety assessment. No Defense is not a substitute for that baseline; the pair experiment provides mechanism evidence rather than a performance ranking against all history-aware guards.

Q6. Harmonic mean and refuse-everything baseline.

A refuse-everything policy has BSU=50% and harmonic mean H=0H=0. Using ASR and BCR as proportions,

H=2(1−ASR)BCR(1−ASR)+BCR.H=\frac{2(1-\mathrm{ASR})\mathrm{BCR}}{(1-\mathrm{ASR})+\mathrm{BCR}}.

Table 3 gives the following HH values (No Defense to TRACE-GUARD, %):

  • DeepSeek-V3.2: 4.5 to 74.4.
  • GPT-5.6-terra: 69.9 to 59.7.
  • Qwen3.7-plus: 69.2 to 31.4.
  • Claude-Sonnet-5: 63.0 to 25.9.

These are calculations from rounded cross-run means, not means of per-run HH. DeepSeek improves, while benign utility loss dominates for the other models. ASR reductions across all four models and improvements in combined utility are therefore distinct conclusions. This reinforces our focus on the proposer and execution interface.

Thank you for the response. However, my main concerns remain unaddressed.

Practicality (W2). The response confirms rather than resolves this concern. By the authors’ own calculation, the harmonic mean of (1 − ASR) and BCR decreases on three of four models (GPT: 69.9 → 59.7; Qwen: 69.2 → 31.4; Claude: 63.0 → 25.9). Moreover, since the proposer and translator are fixed across models (Q1), the collapse of BCR on Qwen and Claude cannot be attributed to proposer quality. It is still unclear why this happens, and no Proposer Only results or failure analysis are provided for these models (Q2).

Full-history guard (Q5). My intention with this question was to verify the design. How does the proposed method compare with simply giving the trajectory so far to an LLM and asking it to judge? The response acknowledges that No Defense is not a substitute for this baseline, but the baseline itself is still not reported.

I therefore maintain my rating.

Reviewer_tiZmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes TRACE-GUARD, a runtime defense that retains structured safety state and evaluates candidate state transitions before dispatching tool actions. It introduces RMSBench, a 300-case executable benchmark combining task-specific environment criteria with runtime decision records. Experiments across four models, state-retention ablations, and 30 controlled-history pairs demonstrate reduced attack success, although benign completion also decreases substantially on some models.

Critical strengths
  • Targeted mechanism evaluation. Controlled-history pairs directly test whether different histories change decisions for identical final actions, providing stronger mechanism evidence than aggregate ASR alone.
  • Outcome–intervention distinction. Combining executable environment checks with runtime records avoids automatically treating task non-completion as successful guard intervention.
Critical weaknesses
  • Insufficient implementation detail and state-reliability evidence. Semantic translation, deterministic extraction, and their reconciliation are underspecified. State is committed before execution confirmation, and object lineage is incomplete. Failed execution or interleaved resources may therefore produce incorrect safety judgments, but these failure modes are not quantified.
  • Limited evaluation independence and criterion validity. Rulebook development used benchmark task specifications, so the main results do not establish development-independent generalization. Moreover, many criteria are acknowledged as proxy or incomplete checks. Consequently, reduced task-criterion ASR does not sufficiently establish effectiveness against stronger harmful objectives in unseen workflows.
Questions for authors
  • Implementation clarification:​ What are the concrete abstraction and reconciliation algorithms, particularly when outputs conflict, fields are missing, or translation fails?
  • Reliability evidence:​ How frequently do failed or partial execution and interleaved resources cause symbolic–physical state disagreement, and what is its safety impact?
  • Evaluation clarification:​ Are development-independent held-out results available? What explains benign completion falling to 18.8% on Qwen and 15.0% on Claude?
1 comment

Thank you for recognizing the controlled-history evaluation and the distinction between environment outcomes and intervention records. Your questions help clarify the implementation and the relationship between policy state and end-to-end execution.

Q1. How are semantic abstraction, deterministic extraction, and reconciliation implemented?

TRACE-GUARD uses field-level reconciliation to combine semantic interpretation with concrete command evidence. The semantic translator generates a structured predicate from the current action and symbolic state. The deterministic extractor identifies operation types, target categories, and risk attributes through command, path, and keyword patterns.

The destructive-operation and external-endpoint flags are each combined by logical OR. When deterministic extraction identifies high sensitivity, sensitivity is set to high. Missing or generic target categories, and operation categories that are missing or classified as other, file_read, or file_write, are supplemented or replaced by deterministic values. Other fields retain their parsed semantic values.

Missing fields receive parser defaults before reconciliation. API or parsing failures trigger configurable retries, followed by deterministic extraction if retries are exhausted. Translation failure therefore follows an explicit fallback path rather than terminating the assessment; this fallback does not automatically imply BLOCK.

Q2. How do execution failures and interleaved objects affect state reliability?

The relevant distinction is between policy state and physical environment state. In Appendix A.1, state commitment records an approved policy transition, while environment checks determine whether the corresponding physical outcome occurred. BLOCK prevents both action dispatch and candidate-state commitment. ALLOW/WARN commits the candidate state before execution feedback returns.

Execution failure therefore requires separate treatment. An approved archive operation may fail after the symbolic state has advanced. Similarly, an operation involving object A should not automatically satisfy a precondition for object B. The controlled-history experiment tests decisions under specified executed prefixes; its results cannot be converted into an incidence rate or safety-impact estimate for execution disagreement.

Our subsequent research direction is to associate policy facts with specific resources and confirm state changes through execution feedback. This would distinguish “archiving was approved” from “the archive of object A was successfully created,” grounding subsequent decisions in more precise execution evidence.

Q3. How should generalization and the decline in benign completion be interpreted?

The rulebook was fixed before the reported evaluation runs and was not updated online during evaluation. The most direct mechanism evidence comes from the controlled-history pairs: with the final action and deterministic predicate fixed, changing the executed prefix allows the stateful guard to distinguish all 30 pairs. This supports explicit retention of historical conditions within the evaluated policy space. Development-independent generalization is a separate property; rule freezing and controlled-history evaluation do not themselves establish it.

For benign tasks, No Defense achieves BCR of 97.1%–99.6%, showing that the original agents complete nearly all tasks. Introducing TRACE-GUARD also introduces stepwise proposal and execution interfaces, which must be considered when interpreting the end-to-end change.

In Table 24, Proposer Only already has BCR of 72.1%, compared with 72.5% for Stateless Guard and 69.6% for Stateful TRACE-GUARD. This makes task decomposition, step formulation, and information transfer important improvement targets. Meanwhile, within the same framework, moving from Stateless to Stateful reduces ASR from 36.8% to 24.2%, while BCR changes from 72.5% to 69.6%, providing evidence of an additional safety benefit from retained state.

We will continue improving the proposer and its execution interface, particularly the preservation of task goals, dependencies, and execution feedback. The larger declines for Qwen and Claude warrant model-specific analysis; the DeepSeek ablation provides a direction for that analysis rather than direct attribution for those models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.