acceptodds
Under review as a conference paper at ICLR 2027

Predictive Trajectory World Models for Process Risk in LLM Agents

Abstract

An LLM agent can reach a correct outcome while following an unreliable execution process. Outcome-level evaluation can therefore miss process failures, while fixed anomaly taxonomies provide task- and dataset-dependent definitions of abnormal behavior. Instead of defining abnormal agent behavior with a fixed taxonomy, we model normal execution and measure how a candidate action is predicted to depart from it. Given a completed trajectory prefix and a candidate next action, we study prospective process risk by predicting before execution whether that action will move the process outside its normal support. Our analysis finds cross-model consistency for normal executions of the same exact task but limited evidence that a whole task category forms one compact process cluster. This motivates task-conditioned local normal support. This also separates executor-specific variation from task information, which remains necessary to define what constitutes normal execution. We align same-task executions across executor models and use retrieved normal context in a trajectory world model that predicts the next latent process state under the candidate action. The world model learns normal process dynamics rather than directly predicting anomaly labels, leaving anomaly supervision to the final onset-aware risk mapping. Predicted future-state and transition deviations then inform an onset-aware risk score focused on the first abnormal step. Experiments support each part of this chain. Geometry analysis shows clear same-task cross-model structure, but does not support a single compact center for a whole task category. Retrieval-conditioned prediction achieves the lowest mean next-state error across cross-model, held-exact-task, and leave-one-subset-out settings. At the 10% development normal-trajectory FAR budget, the final predictive-risk model reaches 56.8% first-alarm exact-onset hit, improving over the matched direct-only baseline by 17.3 percentage points at similar observed FAR. These results show that process risk can be grounded in task-conditioned normal structure and predicted future deviation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Process Risk, Trajectory World Models, Agent Trajectory Auditing, Normal Process Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5h7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies prospective detection of the first anomalous transition in LLM-agent execution. It combines trajectory encoding, task-conditioned normal references, and action-conditioned next-state prediction. Predicted state and transition deviations are fused with direct task–action evidence to estimate risk before execution. Experiments assess next-state prediction, anomaly-onset detection, and computational efficiency across several transfer settings.

Critical strengths
  1. Assessing a candidate action through its predicted consequences relative to normal process support is a meaningful approach to prospective risk estimation.

  2. The reported lightweight auditor achieves a useful performance–cost trade-off. Evaluation across multiple holdout settings and random seeds, with a shared FAR-calibration protocol, provides substantial empirical coverage.

Critical weaknesses
  1. R-A omits trajectory history and normal-reference evidence, while the TS baselines differ in their representation front end and architecture. These comparisons therefore leave the incremental value of explicit next-state prediction unclear. Separately, the value of cross-executor alignment as a distinct contribution is not sufficiently established by the current geometric analysis.

  2. The method introduces several intermediate representations, statistical features, and fusion components, but the numerical encoding of raw events and the construction of the 29-dimensional state remain insufficiently specified. These omissions hinder understanding of the method and make the source of its performance–cost differences from frozen-observer baselines less clear.

Questions for authors
  1. Could the authors clarify which existing results support the added value of explicit next-state prediction over directly using trajectory history and normal-reference information for risk estimation? If available, a direct-risk comparison retaining these information sources would help clarify this point.

  2. In Table 5, distances between different tasks also contract, proportionally more than same-task cross-executor distances. Could the authors provide either a further interpretation of how these geometric results support cross-executor alignment, or, where feasible, a downstream comparison with and without the alignment term on held-out executors at the same FAR budget to support its practical value?

  3. How are raw events encoded for the GRU, and observations mapped to the 29-dimensional state? Could the TS baselines adopt the proposed encoding front end, retaining their downstream decision structures with appropriate retraining, to assess its performance–cost impact? If not, what prevents this adaptation?

1 comment

We thank the reviewer for the constructive comments and recognition of the value of prospective, lightweight process-risk auditing. We have added controlled evaluations, clarified implementation details, and revised the relevant claims in the manuscript.

1. "Added value of explicit next-state prediction … trajectory history and normal-reference information" (W1, Q1)

Our world model learns action-conditioned normal process dynamics and predicts the future structural state induced by a candidate action before execution. By comparing this predicted state and transition with task-conditioned normal support, the risk estimator can incorporate prospective process evidence beyond the observed trajectory history.

To evaluate this contribution, we conducted additional frozen-upstream controlled ablations using the gated fusion mechanism of R-G, progressively incorporating completed history, retrieved normal support, and explicit prediction.

ConfigurationExact-Onset Hit
A: Direct-only38.97%
B: A + History55.79%
C: B + Normal Support58.22%
D: C + Explicit Prediction60.49%

The results show that history and normal support improve exact-onset detection by 16.82 pp and 2.43 pp, respectively. Importantly, adding explicit next-state prediction provides a further 2.27 pp improvement after retaining both information sources.

This directly supports the utility of action-conditioned future-state representations as complementary evidence for prospective risk estimation. Together with the action-conditioning experiments, these findings connect the world model's predictive capability with its contribution to downstream onset detection.

The original 17.3 pp improvement over R-A represents the benefit of the complete integrated framework, while the additional controlled comparison further identifies the incremental contribution of explicit prediction.

Revised: Sec.4.4, Sec.5.4, Appendix C.2.

2. "Geometric results … downstream comparison with and without the alignment term" (W1, Q2)

We thank the reviewer for highlighting the distinction between geometric alignment and downstream detection performance.

Cross-executor alignment serves as an auxiliary representation regularizer, rather than a primary contribution of our framework. Our central contributions lie in task-conditioned local normal support, action-conditioned future-state prediction, and onset-aware risk estimation.

To examine alignment's effectiveness, we conducted an additional downstream ablation on six actual held-out executors (18 paired evaluations), alongside our geometric analysis:

EvaluationMetricResult
GeometrySame-task cross-executor distance0.1286 → 0.0882 (−31.4%)
GeometryDifferent-subset distance0.1941 → 0.1154 (−40.5%)
DownstreamΔ First-Alarm Exact-Onset Hit (λ=0.1 vs. 0)−0.0075
DownstreamSix-source descriptive interval[−0.0213, +0.0012]

These results show that alignment does not yield consistent downstream detection gains. In contrast, task-conditioned normal support provides measurable complementary evidence, as shown in Section 1. Improving alignment while preserving task-specific structure remains future work.

Revised: Sec.5.2, Appendix C.3.

3. "Raw events … 29-dimensional state … TS frontend" (W2, Q3)

(a) "Raw events … GRU … 29-dimensional state?"

Each completed event uses separate 29D action and 29D outcome features, projected to 64D and processed by a 64-unit GRU. The outcome comprises flags (10D), metrics (4D), status (3D), delta (5D), dependency (4D), and side effects (3D). These differ from the learned history state. We also clarify the 304D task/action semantics, lexical hashing, and task/event-compatible normal-reference retrieval. A candidate action is scored before its outcome is observed.

Revised: Sec.4.2–4.3, Appendix B.1.

(b) "TS front end … performance-cost impact?"

We retrained both TS-Traj and TS-Onset decision heads with R-G's structured inputs across 108 contexts:

Shared frontendR-G HitTS-Traj HitTS-Onset Hit
History + Normal Support58.22%54.33%64.21%
+ Explicit Prediction60.49%53.84%62.55%

TS-Onset retains the highest Hit, while R-G exceeds TS-Traj and benefits from prediction. Each adapted TS head has 223,941 trainable parameters (excluding the shared frozen frontend) and omits the original 7B observer.

Under a unified FP32 evaluation of the original frozen-observer configurations, R-G achieves 23.34×/23.43× lower p50 latency than TS-Traj/TS-Onset. Peak allocated GPU memory is 0.0104 GiB versus 31.19 GiB. This supports an accuracy–efficiency trade-off under matched precision.

Revised: Appendix C.5, Sec.5.5, Appendix B.5.

We appreciate the reviewer's suggestions, which have strengthened the evaluation and clarified the scope of our claims.

Reviewer_QehbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work proposes a world modeling strategy for flagging unsafe behaviors in agents. The approach makes use of safe behavior support sets (i.e., safe replays) and similarity to those for risk modeling. The risk model is conditioned on task semantics, trajectory prefix, and candidate next action. The world model predicts the next state (i.e., h_t) given prefix, task, candidate action, which is a Gaussian prediction (so, it predicts the Gaussian parameters of next state). This distribution prediction helps model risk given how closely the distribution tracks a "normal" state.

Critical strengths

The idea seems sound. The idea hinges on the fact that a "risky" state can actually be modeled.

Critical weaknesses

I have two reservations. First, I am unclear if we are actually modeling "risks". It could simply be the case that the world model is using state predictiveness to model anomaly (e.g., [1]).

Second, writing of experimental section. It appears to be filled with information that is often hard to follow.

  1. World Model Anomaly Detection with a Latent Linear Prior, TMLR 2026
Questions for authors
  1. Can the authors show the correlation between state predictability of the world model with risk prediction?

  2. Can the authors discuss [1] as it seems highly relevant to the work?

  3. World Model Anomaly Detection with a Latent Linear Prior, TMLR 2026

1 comment

We thank the reviewer for the constructive feedback and for recognizing the soundness of our approach. We have revised the manuscript to clarify the distinction between state predictability and process risk, provide additional empirical analysis, discuss the suggested related work, and improve the presentation of our experiments.

1. "The correlation between state predictability ... with risk prediction?" (W1, Q1)

Our approach separates future-state prediction from process-risk estimation. The world model learns normal action-conditioned dynamics to predict the next structural state given the completed trajectory and candidate action. The subsequent onset-supervised risk head evaluates the predicted state and transition against task-conditioned normal support, combining this evidence with trajectory history and direct task–action information to estimate the risk of initiating the first abnormal transition.

Importantly, the objective is not to use prediction error directly as a risk score, but to use predicted future-state information to support prospective risk estimation.

Our additional analysis confirms this distinction. Predictive uncertainty correlates strongly with prediction MSE (Spearman ρ = 0.767), whereas uncertainty and MSE correlate negatively with the final risk score (ρ = −0.602 and −0.405). Furthermore, normal-support state and transition deviations achieve higher onset-discrimination AUROC (0.640 and 0.686) than uncertainty or MSE alone (0.329 and 0.439), with the supervised risk head reaching 0.924.

These results show that state predictability alone does not determine onset risk, while predicted state and transition deviations provide meaningful evidence for onset discrimination. By integrating such process information with task-conditioned normal support and onset supervision, our framework enables prospective risk assessment beyond direct task–action scoring.

We have clarified this relationship in the revised Sections 3.2, 4.4, and 5.3, with the additional correlation and discrimination analyses in Appendix C.4 (Table 20). Prediction MSE is computed retrospectively and is never used as a pre-execution input.

2. “Discuss [1] ... highly relevant to the work” (Q2–3)

We appreciate the reference to World Model Anomaly Detection with a Latent Linear Prior (Ikram et al., TMLR 2026). Both works leverage learned world-model dynamics to identify anomalous behavior, but differ in their anomaly objects, supervision, and downstream objectives.

Specifically, Ikram et al. use a latent linear prior to identify unreliable imagined transitions generated by a world model in model-based reinforcement learning, suppressing their influence on value updates and policy learning. In contrast, we assess candidate actions in observed LLM-agent executions before their outcomes are available. Our world model predicts the next structural state under a candidate action, while task-conditioned normal support and first-onset supervision enable the subsequent risk head to estimate whether that action will initiate an abnormal process transition.

We have added this work in the revised Section 2.2 (Related Work).

3. "Experimental section ... hard to follow" (W2)

Thank you for highlighting the presentation issue. We have reorganized Section 5 to clarify the progression from normal process structure (Q1), through action-conditioned next-state prediction (Q2), to prospective onset-risk detection (Q3). We also distinguish the corresponding evaluation settings and metrics, including the external silver world-model evaluation, task-disjoint confirmation, and additional post-submission analyses. These clarifications are reflected in the revised Section 5 and Appendices B–C.

We hope these revisions and additional analyses address the reviewer's concerns and clarify the methodological contribution of our work.

Reviewer_x2egOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on prospective process-risk detection for LLM agents, determining whether a pending action will initiate an abnormal execution process before observing its outcome. In the paper , it combines cross-executor alignment of normal trajectories, retrieval of task-conditioned normal references, and a world model that predicts next-outcome representation. Predicted state and transition deviations are fused with direct task–action evidence using an onset-supervised risk head. Experiments cover held-executor, held-task, and leave-one-subset-out settings. The proposed model improves exact first-alarm onset detection over the direct-only baseline at relatively low normal-trajectory false-alarm rate, while remaining below the strongest large-observer baseline.

Critical strengths
  1. This paper studies a relevant and well defined problem. Evaluating a pending action before execution addresses an important limitation of outcome-only agent evaluation.

  2. The proposed fusion consistently improves over the direct-only baseline across the reported transfer regimes. This supports the usefulness of incorporating process context and normal-reference information into lightweight auditing.

  3. The experiments are complete, which includes multiple seeds, three transfer regimes, paired confidence intervals, threshold sweeps, and label-budget experiments.

Critical weaknesses
  1. R-A receives task/action evidence, whereas R-G additionally receives completed history, retrieved support, and predicted features, so the headline improvement cannot be attributed specifically to prediction.

  2. The paper lacks complete description and definition of the outcome dimensions, task/action semantic features, or retrieval compatibility computation.

  3. Section 5.5 specifies the experiments are done with a single RTX 4090, but Table 12 reports 31.19 GiB of peak allocated GPU memory for the float32 baseline, exceeding the card’s standard 24 GB capacity. Are they under different settings?

  4. The confirmation partition is separated from development for risk-head selection, but it comes from the original validation partition, which is also mentioned in world-model checkpoint selection. Are confirmation trajectories excluded from all upstream selection, rather than only the final risk-head and threshold selection?

Questions for authors
  1. How much of the improvement comes specifically from predicting future states, rather than providing the detector with trajectory history and retrieved normal examples?

  2. Could you provide the definitions of the outcome representation, semantic features, and retrieval score?

  3. What are the main causes of false alarms and missed onsets from the experiment results?

  4. Could you clarify the hardware and memory accounting behind Table 12 and verify the latency advantage using the same numerical precision as the accuracy evaluation?

1 comment

We thank the reviewer for the constructive comments. We have revised the manuscript and added targeted analyses to address all four weaknesses and four questions.

1. “The headline improvement cannot be attributed specifically to prediction” (W1, Q1)

The original 17.3 percentage-point improvement over R-A reflects the benefit of the complete framework, which combines trajectory history, retrieved normal support, and predicted future-state evidence. To quantify their respective contributions, we conducted additional frozen-upstream controlled ablations using matched gated risk heads, progressively introducing these information sources.

ConfigurationExact-Onset Hit
A: Direct-only38.97%
B: A + History55.79%
C: B + Normal Support58.22%
D: C + Explicit Prediction60.49%

The results show that completed history contributes +16.82 pp, while retrieved normal support provides a further +2.43 pp. Crucially, explicit next-state prediction yields an additional +2.27 pp after both information sources are retained.

This comparison isolates the incremental contribution of prediction within the controlled fusion setting. Rather than directly predicting risk, the world model forecasts the structural state induced by a candidate action, allowing the risk estimator to assess prospective state and transition deviations relative to normal support. The additional improvement therefore supports the utility of predicted future-process evidence beyond the observed execution history and retrieved references.

Together with the action-conditioning results, these findings establish both the predictive validity of the world model and the complementary value of its predictions for downstream onset detection. We have clarified that the original +17.3 pp represents the full-system improvement, while the new controlled ablation separately quantifies the incremental benefit of explicit prediction.

Revised: Section 5.4 and Appendix C.2.

2. “Outcome dimensions, task/action semantic features, or retrieval compatibility computation” (W2, Q2)

We distinguish deterministic structural encoding from learned process representation. Each completed event combines 29D action and 29D outcome features into a 58D input, projected to 64D and processed by a 64-unit GRU. The outcome comprises result flags (10D), metrics (4D), status (3D), delta (5D), dependency (4D), and side effects (3D). These coordinates do not manually specify anomaly categories; the GRU learns the temporal process representation.

We also specify the 304D joint task/action semantics, lexical hashing, and retrieval: cosine similarity ranks fit-normal task descriptors; the top 24 are screened for event compatibility, retaining up to eight normal prototypes with specified weights and fallback. Candidate actions are scored before their outcomes are observed.

Revised: Sections 4.2–4.3 and Appendix B.1.

3. “Are confirmation trajectories excluded from all upstream selection…?” (W4)

Historical world-model selection used validation-normal samples, including some later assigned to confirmation; downstream risk-head selection and FAR calibration used development only. We therefore performed development-only upstream revalidation. All 108 historical and 108 retrained world models selected epoch 8, so excluding confirmation did not change the selected epoch. Retraining 108 world models and 216 risk heads yielded 56.72% R-G vs. 39.45% R-A Exact-Onset Hit (+17.28 pp; descriptive 95% CI [16.15, 18.44]), consistent with the original +17.3 pp. These results support the robustness of our findings under stricter upstream selection.

Revised: Sec. 5.1, Appendix C.1.

4. “Main causes of false alarms and missed onsets” (Q3)

We added three cases in Appendix C.6. False alarm: the agent legitimately recovered from unavailable SciPy/NumPy using standard-library tools but was flagged. Missed onset: a market-status task scored just below threshold, producing no alarm. Delayed localization: a Tokyo itinerary task scored just below threshold at onset, triggering one step late. These cases motivate future work on distinguishing legitimate adaptations from abnormal transitions and improving borderline-onset detection.

Revised: Appendix C.6.

5. “Hardware and memory accounting … same numerical precision” (W3, Q4)

We verified that the original experiments used an NVIDIA A100 80GB; the RTX 4090 description was mistakenly carried over from local testing. We additionally measured accuracy and latency under a unified FP32 inference path using frozen models and consistent inputs. R-G achieved 23.34× / 23.43× speedups in p50 latency over TS-Traj / TS-Onset, respectively, while its Exact-Onset Hit was 3.33 / 4.82 pp lower. Peak allocated GPU memory was 0.0104 GiB for R-G versus 31.19 GiB for both TS variants. This confirms a favorable accuracy–efficiency trade-off under matched precision.

Revised: Section 5.5 and Appendix B.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.