acceptodds
Under review as a conference paper at ICLR 2027

HorizonGuard: Runtime Risk Forecasting for LLM Agents

15 views

Abstract

LLM agents increasingly execute long-horizon tool-use trajectories, where safety risks can emerge from user instructions, environment observations, or the agent's own decisions. Existing guard models typically assess isolated prompts, final responses, or completed trajectories, making them ill-suited to intervene before an unsafe action is committed. We introduce HorizonGuard, a runtime risk forecasting model that conditions on the task and the observed trajectory prefix to predict a monotonic cumulative risk curve over the next steps. This formulation naturally enables a sequential first-trigger intervention policy while exposing the forecast horizon and decision threshold as deployment-time controls, allowing a single model to flexibly trade off early warning against false alarms without retraining. To train HorizonGuard, we construct HorizonRisk, a step-level dataset generated through controlled risk injection into real agent trajectories. HorizonRisk covers diverse risk sources, risk types, and execution states, with annotations for both the first risk exposure and the eventual unsafe commitment. We further introduce a runtime-oriented sequential evaluation protocol that measures intervention quality throughout execution rather than after trajectory completion. Experiments across multiple agent benchmarks demonstrate accurate runtime risk forecasting under both loose and strict labeling criteria, robust generalization across diverse trajectory states, and effective adaptation to different deployment risk preferences. These results establish HorizonGuard as a configurable runtime guardrail for proactively detecting and interrupting unsafe agent behavior, advancing agent safety from retrospective trajectory classification to forward-looking runtime risk forecasting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
runtime risk forecasting, LLM agent guard

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_YT2COpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces HorizonGuard, a runtime guard model that forecasts whether an LLM agent will perform an unsafe action within the next several execution steps. Given the task and the observed trajectory prefix, the model predicts a monotonic cumulative risk curve and triggers intervention when the risk exceeds a configurable threshold. The authors also construct HorizonRisk, a step-level dataset generated by injecting different types of risks into agent trajectories and annotating both the first observable risk and the eventual unsafe commitment.

Critical strengths
  • The paper addresses an important practical problem: detecting risks before an agent performs an irreversible unsafe action.

  • The runtime formulation avoids repeatedly executing the task and supports continuous monitoring of a single unfolding trajectory.

  • The distinction between risk exposure, successful defense, and unsafe commitment provides more fine-grained supervision than trajectory-level safety labels.

  • The configurable forecast horizon and intervention threshold provide a useful mechanism for trading off early detection against false alarms without retraining the model.

  • The evaluation includes both an in-domain benchmark and several external benchmarks, together with architectural ablations and analyses of intervention timeliness.

Critical weaknesses
  • The presentation is poor, the text of Table1-4 is too small, figure 3/4 have same problem, it is hard to see the label and axis and follow the image.

  • There are many current works discuss about agent auditing like [1], the author should discuss related work comprehensively.

  • The data is labeled by LLM, author should use human verification to double check the data.

  • The definition in paper is chaos, author use 0, 0.5, 1 as no risk visible, risk exposed, unsafe committed, but author also say output as probability. These label is the risk status defined by human, not probability.

  • Some OOD results are weaker than LLM judge, author should provide more clarification.

  • Paper says the details in Appendix but the appendix is empty.

  • The model name in table is WizardGuard rather HorizonGuard, also there is Table ?? in Line 131

[1] https://arxiv.org/abs/2605.08715.

Questions for authors

I think the idea of the paper may interesting, but because of poor presentation including the inconsistency of the name in paper, the empty part, I think this paper is not ready for ICLR so I decide to give clear reject.

Reviewer_oyCtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HorizonGuard, a runtime safety model that forecasts risks in LLM agents from the task description and observed execution history, aiming to enable intervention before unsafe actions occur. It predicts a monotonic cumulative risk curve over future steps, allowing operators to adjust the forecast horizon and intervention threshold to balance early warnings against false alarms without retraining. The authors also construct HorizonRisk, a step-level dataset distinguishing initial risk exposure from unsafe commitment, and introduce a sequential evaluation protocol for assessing intervention during execution. Experiments demonstrate strong performance on HorizonRisk and promising transfer to external benchmarks, although detection accuracy and false-alarm rates vary across settings.

Critical strengths
  1. The paper addresses an important problem with substantial societal value. As LLM agents increasingly execute commands, access sensitive information, and interact with external systems, unsafe intermediate actions can cause harm before a task is completed. Forecasting such risks and enabling preventive intervention is therefore an important research direction with practical value for privacy protection, system security, and the responsible deployment of autonomous agents.

  2. The runtime forecasting formulation is well motivated. HorizonGuard predicts future risk from observed execution prefixes and supports a sequential first-trigger intervention policy. This formulation directly addresses the need to act before an irreversible unsafe action occurs. The evaluation largely follows this objective by excluding the committing step on HR-Bench, SafetyDrift, and StepShield.

  3. The dataset provides useful distinctions between risk exposure and unsafe behavior. HorizonRisk separately annotates the earliest observable risk and the first unsafe commitment. Its four trajectory states also distinguish benign execution, successful defense, failed defense, and unsafe execution. These distinctions provide richer supervision than trajectory-level binary labels and support analysis of whether suspicious inputs actually lead to unsafe behavior.

  4. The model offers configurable intervention behavior without retraining. The cumulative-risk construction guarantees consistency across forecast horizons, while the operating horizon and decision threshold provide two deployment-time controls. Figure 3 systematically demonstrates how these controls affect precision, recall, and intervention timing, making the trade-off between early warnings and unnecessary interruptions explicit.

Critical weaknesses
  1. The manuscript contains clear writing and presentation issues. The dataset construction section includes an unresolved “Table ??” cross-reference, and Appendix A still contains the template placeholder “You may include other additional sections here.”

  2. The training targets are not fully consistent with the claimed probability interpretation. The paper defines the cumulative output as the probability that an unsafe action will occur within a future horizon. However, it assigns a target of 0.5 to risk exposure without unsafe commitment, including successfully defended trajectories. This is a manually assigned intermediate risk level rather than an empirically justified event probability. Without calibration analysis, the output is more clearly supported as an ordinal risk score than as the stated probability of future unsafe behavior.

  3. The evidence for superior early intervention is incomplete. Although early warning is a central motivation, HorizonGuard’s HR-Bench EIS values are lower than those of several baselines. Under the Standard view, its values are 2.71 and 1.94, compared with 3.86 and 2.42 for Gemini-3.1-Pro-Preview. Moreover, EIS is calculated only on correctly detected unsafe trajectories, so comparisons involve different subsets of examples. Evaluation at matched false-positive rates would better establish whether the proposed forecasting approach offers a genuine timeliness advantage.

Questions for authors

Should the model’s output be interpreted as the probability of future unsafe behavior or as a general risk score?

How does the dataset construction process control annotation errors and potential leakage between related trajectory variants?

Reviewer_t9DcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HorizonGuard, a runtime guard for terminal-based LLM agents. HorizonGuard is a fine-tuned Qwen3.5-4B that reads the agent's trajectory, i.e., the task and the most recent ≤10 steps, separated by special delimiter tokens. Its final-layer embeddings at these delimiter tokens are used to predict a risk curve F(1), …, F(10), where F(k) is the predicted risk that the agent takes an unsafe action within the next k steps. At deployment, the operator chooses a horizon K and a threshold τ, and the guard intervenes the first time F(K) ≥ τ.

The paper also presents HorizonRisk, a synthetic dataset of 8,101 trajectories built from clean traces of the Hermes agent. An LLM injects a risk at some step and rewrites the rest of the trace so that the agent stays benign, defends successfully (Safe-Defended), defends but fails, or acts unsafely. LLMs then mark where the risk appears and where the agent first acts unsafely; these two steps define the training labels (0 before the risk appears, 0.5 once it is visible, 1 once the agent has acted unsafely). Of the 8,101 trajectories, 1,000 are held out as the test set HR-Bench. The guard is trained on the remaining 7,101, plus 5,923 trajectories converted from AgentDoG-Train.

On HR-Bench (Standard view), HorizonGuard reaches 74.8 F1, versus at most 69.5 for seven frontier LLM judges and five guard models. It is also evaluated without further training on R-Judge, SafetyDrift and StepShield. On R-Judge, the paper reports a threshold-averaged F1 of 84.4. On SafetyDrift, it detects 37 of 38 unsafe traces but also flags 16 of 34 safe ones. On StepShield, it reaches 0.86 F1, versus 0.95 for a GPT-4.1-mini judge.

Critical strengths

1. HorizonGuard raises far fewer false alarms on agents that resist a risk.

  • Existing baselines tend to flag trajectories in which a risk appears, even when the agent notices the risk and refuses it. HorizonGuard's training gives this state, a risk that is visible but not acted on, its own label (0.5), and its training data include Safe-Defended trajectories in which the agent resists, so the guard learns to stay quiet while the agent handles the risk. This leads to much better results on HR-Bench under the Standard view (Table 1, 59.60% of Safe-Defended trajectories left unflagged vs at most 41.20% for the LLM judges).

Why it is consequential: This is the paper's main original idea, and it targets a real deployment problem: a guard that keeps interrupting agents that have already resisted an attack stops useful work and is hard to deploy.

Critical weaknesses

1. The contribution is incremental: prior work already covers most of the problem formulation.

  • SafetyDrift estimates the probability of a violation within a fixed horizon, ProbGuard intervenes when a safety probability predicted from a learned Markov model falls below a user-set threshold, and StepShield evaluates how early guards alert on incrementally revealed prefixes.

Why it is consequential: Without the parts that already exist, the contribution is a labeling scheme and a dataset on top of an established formulation.

2. The OOD benchmark results are weaker than the paper claims (Section 5.2).

  • On SafetyDrift (Table 3), HorizonGuard detects slightly more violating traces than the benchmark's own Markov monitor (97.4% vs 94.7%) but flags far more safe traces (false-positive rate 47.1% vs 11.8%), the same rate as the three-step LLM judge (47.1%). On StepShield (Table 4), it beats StaticGuard, ConstraintGuard and HybridGuard but trails the GPT-4.1-mini judge on accuracy (0.85 vs 0.95), F1 (0.86 vs 0.95), EIR₃ (0.87 vs 0.89), false-positive rate (19.4% vs 5.6%) and recall (89.8% vs 95.4%).

Why it is consequential: HR-Bench is built by the same synthesis pipeline as the training data, so the OOD benchmarks are the only test of whether HorizonGuard learned agent risk rather than the generator's style. It does not beat the strongest existing monitor on either SafetyDrift or StepShield, so the paper's gains are shown only on its own synthetic data.

Questions for authors
  1. Can the authors further demonstrate that HorizonGuard works on OOD benchmarks?
  2. Can the authors be more explicit about how HorizonGuard extends beyond SafetyDrift, ProbGuard and StepShield?
Reviewer_FGzvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HorizonGuard is a runtime safety model that predicts the risk of unsafe actions in future steps of LLM agent execution. The authors construct HorizonRisk which is a a step-level dataset generated through controlled risk injection into real agent trajectories, and train a model to forecaset cumulative future risk from observed trajectory prefixes. The ppaer's experiments show that HorizonGuard outperforms several strong and close source LLM judges and guard models on risk detection, while allowing flexiable trade-offs between early intervention and false alarms.

Critical strengths
  1. The paper moves beyond traditional safety detection by forecasting future risks during agent execution. This is an important direction, since early intervention can prevent unsafe actions before they occur.

  2. The authors construct HorizonRisk with step-level risk annotations and evalaute their method across multiple benchmarks. The experiments demonstrate promising performance and show different forecasting horizons and thresholds can balance early intervention against false alarms.

Critical weaknesses
  1. It is unclear how Horizonguard can perdict future unsafe behavior when the observaed trajectory contains no risk signals, especially since risks are injected at later steps.

  2. HorizonGuard's improvements may partly come from learning patterns introduced during synthetic data construction rather than generalizable risk signals, especially given that several strong baselines perform close to random guessing.

Questions for authors
  1. Since risks are injected at selected steps while the preceding trajectory remains unchanged, how can HorizonGuard predict future risks before any risk signal becomes observable? Could the author separartely evaluate forecasting performance before and after suspect step?

I am confused by the input to the model and also data construction. So, for example if we have a trajectory:

Step 1: Safe > Step 2: Safe > Step 3: Risk Exposed > Step 4: Unsafe Action

How can the model tell if the rest of the trajectory is safe or not by only give the Step 1 ans 2 as input to the guard model?

  1. I am concerned that HorizonGuard may learn biases introduced during dataset construction rather than genuinely forecasting future unsafe behavior.

In table 1, several strong baselines achieve accuracy close to 50%, which is roughly random gussing on this balanced benchmark. This makes me wonder whether predicting future unsafe behavior is inherently difficuly when the observed trajectory contains no clear risk signals. Since the dataset is constructed through LLM-based risk injection and annotation, could HorizonGuard's better performance partly come from learning patterns introduced during data construction?

Could the author evaluate HorizonGuard on risk trajectories generated by a different LLM or construction pipeline? This could help clarify whether the model learns generalizable risk signals or dataset-specific patterns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.