acceptodds
Under review as a conference paper at ICLR 2027

SafeAWM: A Verifiable Safety Agent World Model for Proactive Guardrails

Abstract

While autonomous LLM agents increasingly execute environment-altering tool workflows, ensuring operational safety requires anticipating prospective compounding risks without penalizing benign actions. Existing proactive guardrails rely on opaque, continuous latent rollouts that suffer from severe safety aliasing: contextual conflation, where legitimate recovery actions are falsely blocked due to shared execution prefixes with attacks, and temporal inconsistency, where unconstrained latent dynamics yield non-monotonic, contradictory risk estimates. We present SafeAWM, a verifiable safety world model that resolves safety aliasing by decoupling environment dynamics from safety evaluation. To prevent latent shortcut learning, SafeAWM establishes an auditable semantic bottleneck governed by an absolute gradient barrier, grounding risk reasoning strictly on observable operational side effects. Furthermore, SafeAWM formulates multi-horizon hazard forecasting through discrete survival analysis, mathematically guaranteeing structural temporal monotonicity across expanding planning windows. Across SafeAliasBench and three external benchmarks, SafeAWM achieves state-of-the-art proactive safeguarding (e.g., 97.4% F1 on SafeAliasBench and 80.6% F1 on ATBench), provides up to 5 steps of advance hazard forecasting, and slashes false alarms on recovery actions to 2.4% FPR with an inference latency of 36ms.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Aliasing, Agent World Model, Temporal Risk Forecasting, Verifiable Guardrails, Proactive Safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WUnvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies two problems in safety guardrails for tool-using agents: incorrectly rejecting legitimate recovery actions because of preceding malicious behavior, and producing cumulative risk estimates that violate temporal monotonicity across prediction horizons. The proposed method, SafeAWM, matches environment, safety, and action slots; predicts future risks through a 37-dimensional semantic bottleneck representing execution outcomes and state changes; and uses a discrete survival formulation to ensure that cumulative risk does not decrease as the prediction horizon expands. The authors also introduce SafeAliasBench, comprising benign, attack, and recovery trajectories, and report classification performance and inference latency on three external benchmarks.

Critical strengths
  1. The problem is practically relevant. Safety guardrails should detect attacks while allowing agents to perform legitimate recovery actions after suspicious histories. The focus on false positives during recovery is useful for evaluating the trade-off between safety and task utility.

  2. The paired-trajectory evaluation design is promising. Comparing attack and recovery actions under similar or identical histories could help determine whether a model uses the current action and authorization context, rather than relying on dangerous keywords in the history.

  3. The monotone parameterization of cumulative risk is reasonable. The discrete survival formulation guarantees that cumulative risk under the same prefix does not decrease as the prediction horizon expands, and the corresponding algebraic proof is valid. Intermediate predictions with explicit semantics also provide an opportunity for post-hoc auditing.

Critical weaknesses
  1. There is a direct contradiction regarding whether the core components were used in the experiments, making the main contribution difficult to verify. Figure 2 on page 4 explicitly labels the Transition Head as “Optional · not trained this round,” Stage 2 as “Not enabled in current experiments,” and the full world-model extension as absent from the current loss. However, the main text, figure caption, and conclusion present semantic transition prediction and prospective risk modeling as implemented and validated contributions, while page 9 reports prospective forecasting results. The authors do not specify the actual model configuration used for each table and figure. Even if these annotations come from an outdated diagram, the training stages, enabled components, and checkpoints must be clarified. Otherwise, it is unclear whether the experiments evaluate an immediate classifier or the complete proposed method.

  2. The connection between prospective prediction and actual safety intervention is missing. The runtime blocking rule in Equation (6) on page 5 depends only on the immediate violation probability and does not use multi-step future risk. Consequently, the paper does not explain how an action that is currently legitimate but carries substantial future risk would trigger an early intervention. The classification results in Tables 1 and 2 cannot, by themselves, establish that the prospective branch improves runtime safety, particularly when performance gains are attributed directly to that branch. The paper needs an explicit intervention rule based on future risk and a controlled comparison with and without the prospective branch.

  3. The evidence for “five-step advance warning” is insufficient, and the interpretation of Figure 5 conflates different temporal quantities. The horizontal axis denotes the number of steps before the first unsafe action, whereas the caption describes prediction horizons. The former compares different temporal prefixes; the latter expands the prediction horizon under a fixed prefix. Only the latter is covered by the monotonicity proposition. Therefore, the figure does not support the text’s connection between the observed score changes and that proposition. Moreover, higher average risk scores on hazardous trajectories do not necessarily imply more effective early warnings. Without reporting early-warning recall at a fixed false-positive rate, first-alarm lead times, operational thresholds, and benign-trajectory controls, the claimed operational benefit cannot be established.

  4. The claims of “verifiability” and “eliminating shortcut learning” substantially exceed what the method guarantees. The 37-dimensional bottleneck contains model-predicted continuous probabilities rather than independently verified environmental facts. Stop-gradient blocks a particular backward gradient path; it does not ensure accurate intermediate predictions, prevent probability values from carrying additional information, or eliminate reliance on dataset biases. Similarly, orthogonality between representations does not establish semantic disentanglement. The paper does not report prediction quality or calibration for the semantic heads, semantic intervention experiments, or the risk bounds it mentions. The appendix proves an ordering property of the outputs, which does not imply predictive accuracy or system-level safety guarantees.

  5. Essential ablations are missing, leaving the sources of the performance gains unresolved. The method simultaneously introduces paired data, triplet supervision, slot matching, auxiliary state prediction, a semantic bottleneck, and survival modeling, but does not isolate their contributions. In particular, a standard classifier using the same Qwen3-0.6B backbone, training data, splits, and calibration budget is missing. The current results cannot rule out the possibility that most gains come from the data and supervision rather than the proposed architecture. PCA visualizations do not substitute for this evidence. Describing one model using cosine similarity and another using Euclidean distance also does not establish a comparable measure of representation disentanglement.

  6. The data splits and evaluation metrics are insufficiently transparent, and some reported numbers require clarification. The paper claims that group-atomic splitting strictly eliminates leakage, but does not specify the grouping criterion, split sizes, or how templates and underlying tasks are separated. The relationship between Table 2 and TEST-A-500 is unclear, and recovery-specific sample counts and false-positive rates are not reported. It is therefore unclear whether the reported 2.4% FPR applies specifically to recovery actions. In addition, under standard pooled binary classification definitions, the F1, recall, and FPR values in Table 1 imply hazardous-example prevalences of approximately 52.10%, 49.95%, and 52.79% for TRACES, DreamGuard, and SafeAWM, respectively, on SafetyDrift. These differences cannot be explained by rounding alone. Different aggregation procedures or evaluation subsets could explain them, but confusion matrices, a consistent evaluation sample set, and explicit aggregation rules are necessary to substantiate fair comparisons.

  7. The interpretation of the results and the positioning of the contributions are overstated. On SafetyDrift, SafeAWM performs worse than DreamGuard in F1, recall, and FPR, despite the text describing it as “matching DreamGuard.” It also does not achieve the highest F1 among all methods on AgentDojo. The reported results may support advantages on particular benchmarks or a speed–performance trade-off, but not unqualified superiority. Furthermore, monotonicity is a standard structural property of discrete survival models; its practical value here needs to be demonstrated through predictive quality and actual intervention benefits. The current manuscript neither adequately validates these benefits nor establishes the independent contribution of the semantic bottleneck, leaving its central technical claims insufficiently supported.

Questions for authors

See Weaknesses

Reviewer_cnedOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates proactive safety guardrails for tool-use-enabled LLM agents, specifically addressing the issue of "Safety Aliasing"—where legitimate recovery actions are erroneously blocked due to their shared history with malicious attacks—as well as the inconsistency of risk estimation across prediction horizons. The authors propose "SafeAWM," a framework that enables real-time safety detection and forward-looking risk estimation through structured state representation, transition prediction, and survival analysis. The key contributions of this work include:

  1. The introduction of a Structured Safety World Model. Built upon Qwen3-0.6B, this model employs "Slot Cross-Attention" to disentangle environment states, safety constraints, and action representations. It further incorporates a 37-dimensional transition prediction bottleneck and a "Stop-Gradient" mechanism to mitigate representation shortcuts.

  2. The design of monotonic forward-looking risk prediction. By modeling multi-step risk using discrete-time hazard functions, the framework structurally ensures that cumulative risk estimates remain monotonically non-decreasing as the prediction horizon expands.

  3. The development of a benchmark for diagnosing Safety Aliasing. The "SafeAliasBench" dataset comprises 3,518 triplets—consisting of normal, malicious, and recovery sequences that share the same historical prefix—and utilizes a "censoring protocol" to account for unknown execution outcomes.

Experimental results demonstrate F1 scores of 90.30%, 58.33%, and 80.60% on the SafetyDrift, AgentDojo, and ATBench benchmarks, respectively. On SafeAliasBench, the model achieves an F1 score of 0.974 and a False Positive Rate (FPR) of 0.024, with a single-step inference latency of 36 ms.

Critical strengths
  1. Problem definition and diagnostic design with practical significance. The concept of "Safety Aliasing" highlights a flaw in existing guardrails, where legitimate recovery actions can be misidentified as attacks. SafeAliasBench provides a sound evaluation framework for analyzing such failures by isolating action discrepancies through counterfactual triplets that share historical prefixes.

  2. A lightweight architecture with deployment potential. Utilizing a 0.6B-parameter model, SafeAWM achieves a single-step inference latency of 36 ms while delivering competitive detection performance on ATBench, demonstrating a clear efficiency advantage.

  3. Structured modeling and label handling are well-founded. Representing environmental states and safety constraints via independent slots offers a clear design approach for distinguishing between the two; furthermore, treating unknown execution outcomes as censored data avoids the pitfall of assigning unreliable look-ahead labels.

Critical weaknesses
  1. The core claims regarding the world model and proactive forecasting lack direct experimental support. Figure 2 explicitly indicates that the "Transition Head" is untrained, the "World-Model Loss" is excluded from the optimization objective, and "Stage 2" is disabled. Given the decoupled design—separating immediate intervention from the transition bottleneck—the reported detection results appear to stem primarily from the Stage 1 classifier. Figure 5 conflates "Forecasting Horizon" with "Lead Time": if interpreted as a prediction window, the risk trends contradict Proposition 1; if interpreted as the lead time before an event, the proposition is inapplicable. The existing evidence is insufficient to substantiate the paper’s central claims concerning a world model and proactive forecasting.

  2. There are issues regarding the consistency and fairness of the evaluation samples and comparison protocols. Back-calculating from the metrics in Table 1 reveals that most baselines on the "SafetyDrift" benchmark are evaluated against 186 positive samples, whereas "DreamGuard" is evaluated against only 27. The recall figures for "SafeAWM" cannot be replicated using the same number of positive samples, suggesting that the various methods may not have been evaluated on an identical test set. Moreover, while "SafeAWM" undergoes supervised fine-tuning, the baselines are evaluated in a zero-shot setting; additionally, the criteria for action-level versus trajectory-level assessments differ. The absence of standardized samples, consistent training conditions, and confidence intervals makes it difficult to reliably attribute performance gains to specific components.

  3. The study lacks critical ablation analyses, and some of its innovative claims exceed the scope of the provided evidence. The authors fail to ablate components such as "Slot Cross-Attention," "Recovery Triplets," the "Transition Bottleneck," "Stop-Gradient," and the "Hazard Head." There is also no comparison against a standard Qwen3-0.6B classifier fine-tuned on the same dataset. The 37-dimensional bottleneck has not been verified against actual environmental states, undermining the claim that the model is "verifiable." The "Stop-Gradient" mechanism does not guarantee the elimination of information shortcuts, and Proposition 1 is essentially a standard result from survival analysis. Since "SafeAWM" fails to achieve the best F1 score on either "SafetyDrift" or "AgentDojo," the claim that it establishes a new state-of-the-art (SOTA) across the board is unfounded.

Questions for authors
  1. Figure 2 indicates that the components related to the "World Model" are not involved in the current experiment. Which specific, active modules were used to generate the results presented in Tables 1–2 and Figures 4–5? Furthermore, if Stage 2 is not trained, how is the model’s look-ahead prediction capability verified?

  2. Please specify the number of positive and negative samples for each benchmark, and clarify whether all methods were evaluated using the exact same set of test instances. How are action-level predictions translated into trajectory-level determinations? How is fairness ensured when comparing results across different training configurations?

  3. Could you provide a baseline comparison using a standard Qwen3-0.6B classifier trained on the same data, as well as ablation results for each core module? What is the state-transition prediction accuracy at the 37-dimensional bottleneck?

  4. Does the horizontal axis in Figure 5 represent the "Forecasting Horizon" or the "Lead Time"? Could you also include comparisons involving safe trajectories, report the "Lead-Time AUROC" and risk calibration results, and explain the specific verification mechanism for what is labeled as "Verifiable"?

Reviewer_kRMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SafeAWM, a safety framework designed to proactively detect risks in multi-step LLM agent workflows. To address the challenges of contextual conflation and temporal inconsistency in existing proactive guardrails, the authors propose a slot-based attention architecture that disentangles representations from different sources, along with a two-stage training objective that leverages the semantic bottlenecks in this task. Experimental results demonstrate that SafeAWM achieves the highest F1 score and the lowest false-positive rate on the authors' newly curated SafeAliasBench, while also showing performance improvements on three external benchmarks.

Critical strengths

The problem of action mis-rejection and cumulative risk dilution are well-motivated and highly relevant to the safety of modern autonomous agents.

The proposed hazard formulation and the monotonicity property are mathematically well-grounded, providing a theoretical guarantee for the proposed method.

Critical weaknesses

According to Figure 2, the proactive safety world model and transition head, which appear to be central components of the proposed approach, are not enabled in the reported experiments. This substantially weakens the empirical validation of the paper's core contributions.

The proposed training objective combines multiple heterogeneous loss terms, making the overall formulation somewhat complex. More comprehensive ablation studies are needed to disentangle the contributions of individual loss components and assess their respective impacts on the final performance.

Questions for authors
  1. Could the authors clarify which components of SafeAWM are actually implemented and used to produce the reported results? For those components not enabled, are there some reasons for it?

  2. Could the authors provide some ablation studies isolating the effects of slot-based representation disentanglement, cross-attention, counterfactual triplet supervision, and the semantic bottleneck?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.